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Executive  Summary 


1.  Background.  Approximately  a  decade  ago,  the  Military  Operations  Research  Society 
(MORS)  sponsored  a  series  of  three  workshops  under  the  rubric  of  Simulation 
Technology  1997  (SIMTECH  97).  Those  workshops  focused  on  identifying  and 
satisfying  the  simulation  technology  needs  of  the  analyst  in  the  late  1990s.  Ultimately, 
that  activity  culminated  with  a  set  of  findings  and  recommendations  on  four  major 
themes:  lifecycle  management  for  Modeling  and  Simulation  (M&S);  a  workstation  for 
the  analyst;  dealing  with  “soft  factors”  (e.g.,  cognitive  factors,  performance  modulators) 
in  M&S;  and  responding  to  M&S’s  needs  for  data.  In  1997,  several  of  the  original 
organizers  of  SIMTECH  97  believed  that  it  was  an  appropriate  time  to  re-assess  the 
results  of  the  prior  workshops  and  to  look  ten  years  into  the  future. 

2.  Goals  and  Objectives.  The  overarching  goal  of  this  new  series  of  workshops  was  to 
promote  more  effective  dialogue  between  the  M&S  technology  community  and  an 
expanded  set  of  users  of  M&S:  analysts,  acquirers  and  educators  and  trainers. 

Consistent  with  this  goal,  four  subordinate  objectives  were  identified: 

•  Review  and  assess  the  findings  and  recommendations  from  SIMTECH  97 ; 

•  Identify  and  prioritize  the  needs  of  the  users  of  military  M&S; 

•  Assess  the  probable  evolution  of  M&S  technology  over  the  next  decade;  and, 

•  Identify  opportunities  to  address  user  needs. 

3.  Approach.  To  satisfy  these  goals  and  objectives,  two  workshops  were  convened. 

The  first  workshop  was  conducted  at  GRCI,  Tysons  Comer,  VA,  on  16-18  December 
1997.  To  symbolize  the  objectives  of  the  workshop,  the  organizers  adopted  the  JANUS 
icon;  a  archetypical  GI  looking  to  the  past  and  a  cyber  warrior  looking  to  the  future. 
Consistent  with  those  dual  perspectives,  Ed  Brady,  FS,  Chair  of  SIMTECH  97,  provided 
a  keynote  address  entitled  “Then  &  Now.”  That  was  followed  by  retrospective 
assessments  by  working  groups  organized  around  the  four  major  themes  that  were 
addressed  in  SIMTECH  97.  To  set  the  stage  for  assessments  of  the  future,  the  second  day 
began  with  plenary  presentations  on  M&S  plans  and  programs  in  the  areas  of  education 
and  training  (Lou  Finch,  DUSD  (Readiness),  OSD),  analysis  (Eric  Coulter,  PA&E,  OSD) 
and  acquisition  (Robin  Frost,  DTSE&E,  OSD).  In  addition,  Paul  Davis,  RAND,  provided 
perspectives  on  the  future  of  M&S  technology  by  summarizing  the  major  findings  and 
recommendations  from  a  recent  National  Research  Council  study.  Drawing  on  these 
presentations  and  the  lessons  learned  from  the  retrospective  assessments,  the  participants 
were  reorganized  into  parallel  clusters  of  M&S  users  and  technologists.  The  users 
identified  and  prioritized  the  M&S  needs  of  analysts,  acquirers  and  educators  and 
trainers.  The  technologists  formulated  a  taxonomy  for  M&S  technology  and,  within  that 
context,  forecast  conservative  and  aggressive  estimates  for  the  state  of  M&S  technology 
by  2007. 

The  second  workshop  was  conducted  at  the  Institute  for  Defense  Analyses  (IDA), 
Alexandria,  VA,  on  18-20  August  1998.  The  workshop  began  with  a  keynote 
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presentation  by  General  Larry  Welch  (USAF,  retired),  President,  IDA,  entitled  Some 
DSB  Task  Force  Perspectives  on  Simulation  for  Innovation.”  His  remarks  were  followed 
by  a  series  of  plenary  presentations  on  emerging  revolutions  in  education  and  training 
(VADM  Patricia  Tracey,  N7,  USN),  analysis  (Vince  Roske,  Joint  Staff),  acquisition 
(Patricia  Sanders,  DTSE&E,  OSD)  and  a  cross-cutting  presentation  on  M&S  as  an 
enabler  of  these  revolutions  (Vem  Bettencourt,  FS,  AMSO,  USA).  Drawing  on  these 
inputs,  hybrid  working  groups  of  M&S  users  and  technologists  refined  their  products 
from  Workshop  I.  Subsequently,  after  a  sequence  of  M&S  technology  presentations, 
these  hybrid  working  groups  identified  a  comprehensive  set  of  shortfalls  (subsuming 
policy,  management  and  technology)  and  formulated  recommendations  to  ameliorate 
them.  These  working  groups  completed  their  deliberations  by  performing  sensitivity 
analyses  to  explore  the  robustness  of  their  findings  and  recommendations  to  changes  in 
key  underlying  assumptions  (e.g.,  availability  of  resources). 

4.  Products.  Four  major  classes  of  products  were  developed  during  the  course  of  the 
workshops.  First,  a  set  of  lessons  learned  from  SIMTECH  97  were  developed.  These 
include  the  importance  of  documenting  assumptions  when  dealing  with  a  volatile  world 
and  the  need  to  formulate  technology  projection  envelopes  (vice  point  predictions).  In 
many  instances,  the  participants  validated  that  key  SIMTECH  97  recommendations  that 
were  not  fully  acted  on  during  the  past  decade  (e.g.,  the  need  for  focused  programs  on 
“soft  factors”  and  data)  remain  as  important  priorities. 

Second,  several  key  assumptions/drivers  for  the  next  decade  were  identified.  These 
include  trends  in  missions  (e.g.,  emerging  asymmetric  threats),  strategy/operational 
concepts  (e.g.,  continued  commitment  to  the  basic  precepts  underlying  Joint  Vision 
2010),  institutions  (e.g.,  forces  will  be  restructured  and  downsizing  of  DoD  will 
continue),  systems  (e.g.,  heightened  interest  in  treating  the  “soldier  as  a  system”),  training 
(e.g.,  increasing  importance  of  advanced  distributed  learning),  commercial  products  (e.g., 
increasing  importance  of  commercial  information  technology  products  for  DoD)  and 
technology  (e.g.,  information  will  be  increasingly  globalized  and  accessible). 

Third,  M&S  technology  projections  were  developed  for  the  next  decade.  As  a  basis  for 
these  projections,  a  taxonomy  was  developed  that  can  be  depicted  as  a  jig  saw  puzzle 
with  four  interlinking  pieces:  modeling  methodology  (i.e.,  the  theories,  processes, 
algorithms  and  information  that  support  the  conceptualization  of  a  model);  development 
methodology  (i.e.,  the  tools,  techniques  and  software  used  in  architecting,  designing  and 
implementing  a  model);  computation  and  communications  technology  (i.e.,  the  platform 
the  M&S  application  is  hosted  on,  how  it  connects  to  other  M&S  applications  and  how 
M&S  application  developers  and  users  connect  to  one  another);  and  data  and  information 
technology  (i.e.,  the  processes  and  tools  needed  to  acquire  and  transform  data  and 
information).  For  each  of  these  areas,  technology  projections  were  made  under 
conservative  assumptions  (e.g.,  continuation  of  current  investment  priorities)  and 
aggressive  assumptions  (e.g.,  substantial  increase  in  priority  with  the  subsequent 
likelihood  of  a  breakthrough). 
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Fourth,  for  the  functional  areas  of  analysis,  acquisition  and  education  and  training,  top- 
down  assessments  were  performed.  These  include  an  articulation  of  a  vision  for  the 
functional  area;  an  identification  of  associated  needs  (in  policy,  management  and 
technology);  a  characterization  of  perceived  shortfalls  (e.g.,  an  identification  of  cases 
where  technology  needs  exceeded  aggressive  projections  (assessed  as  “red”)  and  cases 
where  technology  needs  fell  between  conservative  and  aggressive  projections  (assessed 
as  “amber”));  a  set  of  recommendations  to  ameliorate  perceived  shortfalls;  and  sensitivity 
assessments  to  establish  the  robustness  of  the  recommendations. 

5.  Selected  Findings.  Each  of  the  plenary  speakers  at  the  second  workshop  identified 
M&S  as  a  key  enabler  to  promote  revolutions  in  analysis,  acquisition  and  education  and 
training.  This  hypothesis  was  validated  by  the  working  groups. 

Several  of  the  plenary  speakers  observed  that  many  of  the  obstacles  to  these  revolutions 
are  cultural  in  nature.  Among  the  more  important  cultural  obstacles  identified  were 
institutional  barriers  (e.g.,  the  need  to  go  from  “stovepiped”  organizations  to  more 
collaborative  organizations  that  would  promote  the  more  efficient  sharing  of  tools,  data 
and  expertise);  modeling  and  simulation  barriers  (e.g.,  transitioning  from  the  inflexibility 
of  current  M&S  to  more  flexible  M&S  to  explore  easily  new  operational  concepts, 
doctrines,  procedures,  and  the  human  dimension);  and  process  barriers  (e.g.,  transition 
from  the  use  of  a  few,  “blessed”  scenarios  to  a  full  range  of  scenarios  that  span  the 
mission  space).  Again,  these  observations  were  extended  and  validated  by  the  working 
groups. 

From  a  technology  perspective,  the  working  groups  concluded  that  the  most  significant 
shortfalls  were  projected  to  occur  in  modeling  methodology  (i.e.,  adequate  representation 
of  key  cognitive  factors,  performance  modulators  and  computer  generated  forces); 
development  methodology  (i.e.,  system  architecture/engineering;  system  composability, 
scalability;  and  standards  for  design,  interoperability  and  reuse);  and  data/information 
understanding  (i.e.,  tools  for  dealing  with  data  acquisition,  transformation,  and  access; 
tools  to  support  collaboration).  In  almost  all  cases,  these  projected  technology  shortfalls 
cut  across  individual  functional  areas.  It  is  notable  that  each  functional  working  group 
also  opined  that  commercial  developments  in  communications  and  computing  would 
probably  not  constrain  M&S  applications,  with  the  exception  of  security  needs. 

6.  Recommendations.  Consistent  with  the  selected  findings,  the  following 
recommendations  have  been  formulated. 

6.1  Institutional  Initiatives.  To  facilitate  the  development  of  a  better  balanced  M&S 
Science  and  Technology  (S&T)  investment  strategy,  it  is  necessary  to  develop  a  clear, 
comprehensive  audit  trail  for  current  M&S  S&T  programs  and  plans. 

To  promote  needed  community  sharing  of  tools,  data  and  expertise,  organizational  focal 
points  are  required  for  Simulation  Based  Acquisition  (SBA)  and  Advanced  Distributed 
Learning  (ADL).  These  organizations  should  champion  these  processes,  promote  pilot 
programs,  monitor  commercial  developments,  begin  to  establish  the  community 
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infrastructure  needed  to  “boot  strap”  the  processes  and  assure  the  full  scope  of  cross¬ 
cutting  activities  are  undertaken  (e.g.,  ensure  that  education  and  training  needs  are  treated 
adequately  in  SBA). 

6.2  Program  Initiatives.  An  expanded  family  of  flexible,  readily  tailorable  M&S  is 
needed  to  address  many  user  needs.  Although  on-going  monolithic  model  developments 
(e.g.,  JWARS,  JSIMS)  will  probably  prove  to  be  central  elements  of  this  family,  they  will 
almost  certainly  not  be  sufficient  to  satisfy  the  needs  of  all  users.  To  complement  them, 
“boutique”  models  are  needed  that  address  all  aspects  of  the  expanding  mission  space 
(e.g.,  asymmetric  conflict;  new  operational  concepts).  These  include  system  dynamics 
models  (to  provide  the  ability  to  quickly  scan  and  pre-filter  scenario  space)  through 
virtual  M&S  (to  capture  the  effects  of  distributed  teams  of  people  under  stress).  In 
particular,  the  demands  of  advanced  warfighting  experiments  mandate  new  classes  of 
M&S,  which  are  sufficiently  flexible  to  explore  easily  new  operational  concepts,  and 
companion  education  and  training  experimentation  plans  to  address  the  subjects’  needs 
for  associated  training. 

6.3  Research  Initiatives.  To  redress  identified  M&S  technology  shortfalls  that  affect  all 
users  of  M&S,  undertake  organized  research  programs  in  “soft  factors”  (e.g.,  cognitive 
factors,  performance  modulators,  computer  generated  forces),  data  (e.g.,  tools  to  capture, 
transform,  and  access  data)  and  selected  subjects  in  fundamental/applied  research  (e.g., 
agent-based  modeling,  search  and  model  building;  variable  structure  simulation;  multi¬ 
resolution  modeling;  role  of  interactiveness  in  discovery  and  analysis).  Mechanisms 
must  also  be  established  to  ensure  that  the  results  of  these  research  programs  are  injected 
into  the  practice  of  M&S. 

7.  Summary.  A  great  deal  of  progress  in  military  M&S  has  been  made  over  the  past 
decade,  due  in  no  small  measure  to  the  efforts  of  the  Defense  Modeling  and  Simulation 
Office  (DMSO).  However,  M&S  is  perceived  to  be  an  enabler  for  incipient  revolutions 
in  analysis,  acquisition  and  education  and  training.  It  will  require  institutional  initiatives, 
an  evolving  family  of  flexible  M&S  and  a  focused  research  program  to  realize  those 
revolutions. 
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Main  Report 


1.  Background.  Approximately  a  decade  ago,  the  Military  Operations  Research  Society 
(MORS)  sponsored  a  series  of  three  workshops  under  the  rubric  of  Simulation  Technology  1997 
(SIMTECH  97).  Those  workshops  focused  on  identifying  and  satisfying  the  simulation 
technology  needs  of  the  analyst  in  the  late  1990s.  Ultimately,  that  activity  culminated  with  a  set 
of  findings  and  recommendations  on  four  major  themes:  lifecycle  management  for  Modeling  and 
Simulation  (M&S);  a  workstation  for  the  analyst;  dealing  with  “soft  factors”  (e.g.,  cognitive 
factors,  performance  modulators)  in  M&S;  and  responding  to  M&S’s  needs  for  data.  In  1997, 
several  of  the  original  organizers  of  SIMTECH  97  believed  that  it  was  an  appropriate  time  to  re¬ 
assess  the  results  of  the  prior  workshops  and  to  look  ten  years  into  the  future. 

2.  Goals  and  Objectives.  The  overarching  goal  of  this  new  series  of  workshops  was  to  promote 
more  effective  dialogue  between  the  M&S  technology  community  and  an  expanded  set  of  users 
of  M&S:  analysts,  acquirers  and  educators  and  trainers. 

Consistent  with  this  goal,  four  subordinate  objectives  were  identified:  review  and  assess  the 
findings  and  recommendations  from  SIMTECH  97;  identify  and  prioritize  the  needs  of  the  users 
of  military  M&S;  assess  the  probable  evolution  of  M&S  technology  over  the  next  decade;  and 
identify  opportunities  to  address  user  needs. 

3.  Approach.  To  satisfy  these  goals  and  objectives,  two  workshops  were  convened.  The 
individuals  who  organized  and  executed  these  workshops  are  identified  in  Appendix  D.2. 

The  first  workshop  was  conducted  at  GRCI,  Tysons  Comer,  VA,  on  16-18  December  1997.  The 
overall  approach  to  this  workshop  is  illustrated  in  Figure  1. 


Figurel.  Approach  to  Workshop  I 

To  symbolize  the  objectives  of  the  workshop,  the  organizers  adopted  the  JANUS  icon;  an 
archetypal  GI  looking  to  the  past  and  a  cyber  warrior  looking  to  the  future.  Consistent  with  those 
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dual  perspectives,  Ed  Brady,  FS,  Chair  of  SIMTECH  97,  provided  a  keynote  address  entitled 
“Then  and  Now.  ”  That  was  followed  by  retrospective  assessments  by  working  groups  organized 
around  the  four  major  themes  that  were  addressed  in  SIMTECH  97.  To  set  the  stage  for 
assessments  of  the  future,  the  second  day  began  with  plenary  presentations  on  M&S  plans  and 
programs1  in  the  areas  of  education  and  training  (Lou  Finch,  DUSD  (Readiness),  OSD),  analysis 
(Eric  Coulter,  PA&E,  OSD)  and  acquisition  (Robin  Frost,  DTSE&E,  OSD).  In  addition,  Paul 
Davis,  RAND,  provided  perspectives  on  the  future  of  M&S  technology  by  summarizing  the 
major  findings  and  recommendations  from  a  recent  National  Research  Council  (NRC)  study. 
Drawing  on  these  presentations  and  the  lessons  learned  from  the  retrospective  assessments,  the 
participants  were  reorganized  into  parallel  clusters  of  M&S  users  and  technologists.  The  users 
identified  and  prioritized  the  M&S  needs  of  analysts,  acquirers  and  educators  and  trainers.  The 
technologists  formulated  a  taxonomy  for  M&S  technology  and,  within  that  context,  forecast 
conservative  and  aggressive  estimates  for  the  state  of  M&S  technology  by  2007. 

The  second  workshop  was  conducted  at  the  Institute  for  Defense  Analyses  (IDA),  Alexandria, 
VA,  on  18-20  August  1998.  The  overall  approach  to  this  workshop  is  illustrated  in  Figure  2. 


Technology 

Views 

Refined 

Technology 

Assessment 

Refined 

User 

Needs 

«- 

Visions 

Figure  2.  Approach  to  Workshop  H 

The  workshop  began  with  a  keynote  presentation  by  General  Larry  Welch  (USAF,  retired). 
President,  IDA,  entitled  “Some  DSB  Task  Force  Perspectives  on  Simulation  for  Innovation.  ” 
His  presentation  focused  on  the  cultural  changes  that  are  required  in  three  dimensions: 
institutional,  M&S  and  mission  orientation.  His  remarks  were  followed  by  a  series  of  plenary 
presentations  on  emerging  revolutions  in  education  and  training  (VADM  Patricia  Tracey,  N7, 
USN),  analysis  (Vince  Roske,  Joint  Staff),  acquisition  (Patricia  Sanders,  DTSE&E,  OSD)  and  a 
cross-cutting  presentation  on  M&S  as  an  enabler  of  these  revolutions  (Vem  Bettencourt,  FS, 
AMSO,  USA).  These  briefings  were  followed  by  a  sequence  of  M&S  technology  presentations, 


1  The  vugraphs  employed  by  the  plenary  speakers  at  both  workshops  are  included  in  Appendices  E  to  G  of  these 
Proceedings.  In  addition,  the  affiliations  cited  for  participants  reflects  their  positions  when  the  workshops  were 
convened. 
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including  an  overview  of  the  DoD  M&S  Science  and  Technology  (S&T)  program  by  Gary 
Yerace,  DMSO,  and  various  perspectives  on  trends  in  M&S  technologies  (i.e.,  Dr  Paul  Davis  on 
Modeling  Methodologies,  Dr  Richard  Nance,  VPI  &  SU,  on  Design  Methodologies,  Dr  Richard 
Weatherly,  MITRE,  on  Computer  and  Communications  Capability  and  Dr  Steve  Bankes,  RAND, 
on  Information  Understanding).  Drawing  on  these  inputs,  hybrid  working  groups  of  M&S  users 
and  technologists  refined  their  products  from  Workshop  I.  Subsequently,  these  hybrid  working 
groups  identified  a  comprehensive  set  of  shortfalls  (subsuming  policy,  management  and 
technology)  and  formulated  recommendations  to  ameliorate  them.  These  working  groups 
completed  their  deliberations  by  performing  sensitivity  analyses  to  explore  the  robustness  of  their 
findings  and  recommendations  to  changes  in  key  underlying  assumptions  (e.g.,  availability  of 
resources). 

4.  Products.  Four  major  classes  of  products  were  developed  during  the  course  of  the 
workshops:  lessons  learned  from  SIMTECH  97;  key  assumptions/drivers  for  the  next  decade; 
M&S  technology  projections;  and  top-down  assessments  from  the  perspectives  of  the  analyst, 
acquirer  and  educator  and  trainer. 

4.1  Lessons  Learned  from  SIMTECH  97.  At  the  first  workshop,  the  participants  were  charged 
with  reviewing  SIMTECH  97  and  characterizing  its  accomplishments  and  shortcomings.  It  was 
concluded  that  SIMTECH  97  had  several  notable  successes.  First,  it  catalyzed  Service 
investments  in  M&S.  Subsequently,  it  was  a  source  for  the  newly  created  Defense  Modeling  and 
Simulation  Office  (DMSO),  which  drew  upon  the  workshop’s  products  and  participants  to  create 
and  prioritize  its  Science  and  Technology  agenda.  More  broadly,  it  espoused  concepts  which 
helped  to  shape  community  perspectives  on  M&S  (e.g.,  it  championed  the  concept  of 
evolutionary  development  of  M&S  vice  the  classic  waterfall  development  model). 

In  retrospect,  there  were  several  areas  in  which  SIMTECH  97  was  less  successful.  First,  while 
looking  out  a  decade,  it  made  implicit  assumptions  about  constancy  which  turned  out  to  be  false 
(e.g.,  it  failed  to  recognize  that  the  Cold  War  was  on  the  verge  of  ending).  Second,  it  failed  to 
anticipate  several  of  the  emerging  technology  trends  that  have  had  a  profound  effect  upon  the 
M&S  community  (e.g.,  the  exponential  growth  in  the  use  of  the  Internet).  Third,  in  several  areas, 
it  gave  emphasis  to  issues  which  turned  out  to  be  of  secondary  interest  (e.g.,  the  role  of  massively 
parallel  systems).  Finally,  it  formulated  several  recommendations  which  were  never  acted  upon 
adequately  and  are  still  valid  today  (e.g.,  undertake  a  research  program  on  “Soft  Factors;”  initiate 
a  program  to  acquire,  transform,  verify,  validate  and  certify  data  needed  by  the  M&S 
community). 

Based  upon  this  assessment,  the  workshop  participants  formulated  the  following 
recommendations: 

4.1.1  Document  key  assumptions  explicitly.  Any  attempt  to  forecast  the  future  involves  a  host 
of  assumptions.  These  assumptions  should  be  made  clear  to  the  recipient  of  the  report  so  that 
they  are  aware  of  them  and  can  assess  the  potential  impact  if  they  turn  out  to  be  incorrect. 
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4.1.2  Perform  sensitivity  explorations.  In  view  of  the  volatility  of  the  key  factors  that 
influence  the  nature  of  the  problem,  it  is  important  to  perform  sensitivity  assessments.  This 
should  take  the  form  of  explorations  of  the  robustness  of  findings  and  recommendations  to 
changes  in  key  assumptions. 

4.1.3  Don’t  generate  “point”  solutions.  In  general,  the  participants  concluded  that  it  was  better 
to  be  approximately  right  than  precisely  wrong.  Thus,  it  is  recommended  that  future  projections 
be  treated  using  bounding  arguments  (e.g.,  formulate  “conservative”  and  “aggressive”  estimates). 

4.1.4  Work  assiduously  to  disseminate  the  workshop  findings.  In  order  to  enhance  the 
likelihood  that  the  recommendations  of  a  workshop  are  implemented,  it  was  strongly 
recommended  that  the  workshop  leadership  seek  to  identify  and  work  with  a  “champion.”  In  the 
area  of  M&S,  DMSO  is  an  excellent  candidate  for  that  role.  In  addition,  many  of  the 
recommendations  involve  communities  that  transcend  the  normal  MORS  membership.  Thus,  it 
was  recommended  that  steps  be  taken  to  spread  the  word  beyond  the  “usual  (MORS)  suspects.” 

4.1.5  Continue  to  push  for  action  on  key  residual  recommendations.  Several  of  the 
participants  observed  that  several  of  the  recommendations  made  by  SIMTECH  97  are  still 
germane  today.  Thus,  substantively,  they  recommended  that  efforts  be  made  to  implement  two 
of  the  key  SIMTECH  97  recommendations:  undertake  a  coordinated  “Soft  Factors”  research 
program  and  initiate  a  coordinated  program  to  acquire,  transform,  verify,  validate  and  certify  data 
needed  by  the  M&S  community. 

4.2  Key  Assumptions  and  Sensitivity  Excursions.  Consistent  with  the  recommendation  from 
Workshop  I,  the  workshop  participants  identified  key  assumptions/drivers  for  the  next  decade  as 
a  basis  for  their  deliberations.  In  addition,  they  identified  a  set  of  excursions  from  these 
assumptions  to  support  sensitivity  assessments  (i.e.,  to  explore  the  robustness  of  their  preliminary 
findings  and  recommendations  to  hypothesized  variations  in  these  assumptions).  The  following 
section  identifies  and  briefly  characterizes  those  assumptions  and  sensitivity  excursions  (see 
Figure  3). 


Figure  3.  Macro  Environmental  Assumptions  for  2007 
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4.2.1  Mission.  The  workshop  participants  concluded  that  the  likelihood  of  a  major  theater  war 
within  the  next  decade  is  low.  However,  it  was  observed  that  the  mission  space  and  its  diversity 
will  probably  increase.  This  will  probably  be  manifested  in  increases  in  asymmetric  warfare, 
operations  other  than  war,  urban  warfare,  use  of  weapons  of  mass  destruction,  cyberwarfare,  and 
attacks  against  critical  infrastructures.  As  an  excursion  for  sensitivity  analyses,  it  was  postulated 
that  a  peer  competitor  would  emerge  prior  to  2007. 

4.2.2  Strategy.  The  workshop  participants  projected  that  the  US  will  continue  to  pursue  the 
“shape,  respond,  prepare”  strategy  articulated  in  the  recent  Quadrennial  Defense  Review  (QDR). 

In  addition,  the  basic  precepts  underlying  Joint  Vision  2010  will  remain  in  effect.  For  example, 
there  will  be  continuing  interest  in  formulating/experimenting  with  innovative  warfare  concepts 
(e.g.,  new  doctrine,  concepts  of  operations)  and  efforts  will  continue  to  enhance  the  quality  of 
warfighter  information  and  decrease  the  time  needed  to  make  effective  decisions.  Consistent 
with  Joint  Vision  2010,  it  was  projected  that  the  US  military  will  increasingly  be  in  joint 
operations  with  international  coalitions  of  the  willing,  non-DoD  US  Government  agencies  and 
Non-Governmental  Organizations  (NGOs).  Operationally,  it  was  hypothesized  that  distributed 
operations  (e.g.,  reachback)  will  increase.  As  an  excursion  for  sensitivity  analyses,  two 
alternative  outcomes  for  Joint  Vision  2010  were  hypothesized:  it  is  perceived  as  successful  (i.e., 
new  joint  concepts  of  operation  are  adopted)  or  it  is  perceived  as  unsuccessful  (i.e.,  return  to 
more  deliberate,  mass-oriented  operations). 

4.2.3  Institutional.  The  workshop  participants  identified  five  key  institutional  trends  for  the 
next  decade.  First,  they  projected  that  the  “top  line”  defense  budget  will  be  relatively  flat. 

Hence,  the  primary  opportunity  for  procurement  relief  will  be  to  reduce  Operations  and  Support 
(O&S)  costs.  In  addition,  there  will  be  continual  pressure  to  achieve  greater  efficiency  in  the 
DoD  acquisition  process.  Second,  military  forces  will  be  restructured  and  downsizing  of  DoD 
will  continue.  Third,  CINC  responsibilities  will  continue  to  evolve.  As  one  example,  the  roles 
and  missions  of  US  ACOM  will  continue  to  change  as  it  begins  to  take  a  major  role  in  joint 
experimentation.  Fourth,  classical  institutional  “stovepipes”  will  increasingly  merge.  As 
examples,  it  is  anticipated  that  communities  like  military  operations  and  training,  M&S  and 
Command  and  Control  (C2)  will  have  increasing  confluence  of  interests.  Fifth,  it  is  anticipated 
that  senior  level  decision  makers  will  become  increasingly  comfortable  with  M&S  and  its 
products.  As  an  excursion  for  sensitivity  analyses,  the  workshop  participants  considered 
alternative  futures  in  which  “top  line”  defense  budgets  and  M&S  expenditures  might  deviate 
+/-  20%  from  the  baseline. 

4.2.4  Systems.  The  workshop  participants  identified  four  macro-trends  in  the  area  of  military 
systems.  First,  it  is  projected  that  DoD  will  continue  to  emphasize  the  acquisition  of  relatively 
few,  high  technology,  quality  systems.  This  trend  will  be  driven,  in  part,  by  the  desire  to  reduce 
manpower  needs.  Second,  it  is  anticipated  that  there  will  be  a  broader  diversity  of  systems  on  the 
battlefield.  For  example,  Blue  forces  are  likely  to  be  equipped  with  non-lethal  systems  and  a 
broader  array  of  unmanned  systems.  From  a  Red  perspective,  it  is  more  likely  that  adversaries 
will  possess  weapons  of  mass  destruction  and  disruption.  Third,  there  will  be  a  trend  towards 
more  modular,  “just  in  time,”  tailored  buying  for  selected  systems.  This  trend  will  probably 
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depend  on  the  success  of  the  concept  of  Simulation  Based  Acquisition  (SBA).  Finally,  there  will 
be  heightened  interest  in  treating  the  “soldier  as  a  system.”  Two  cases  were  selected  as 
excursions  from  this  baseline.  First,  a  variant  was  postulated  in  which  limited  acquisition 
resources  prompt  the  DoD  to  plan  to  retain  systems  for  50%  longer  than  existing  systems. 
Second,  a  variant  was  projected  in  which  DoD  shifts  to  an  acquisition  strategy  that  emphasizes 
more,  cheaper  (possibly  lower  tech)  systems. 

4.2.5  Training.  The  workshop  participants  observed  that  the  changing  environment  implies 
new  training  requirements  and  constraints.  In  the  former  case,  there  is  a  need  for  lifelong 
leaming/training;  the  ability  to  train  effectively  with  others;  more  embedded  training;  more 
realistic  training  for  stressful  environments  like  night,  adverse  weather,  complex  terrain,  and 
nuclear,  biological  or  chemical  conditions.  In  the  latter  case,  training  is  likely  to  be  constrained 
by  the  limited  training  time  available.  Alternatively,  new  technologies  will  provide  new  training 
opportunities  (e.g.,  advanced  distributed  learning,  continuous  training  opportunities).  In  view  of 
the  limited  time  available,  the  workshop  participants  opted  not  to  consider  alternative  training 
assumptions. 

4.2.6  Commercial.  The  workshop  participants  forecast  that  DoD  will  be  increasingly  reliant  on 
commercial  industry  for  information  systems  (e.g.,  proliferated  GPS,  cellular  phones).  This 
commercial  technology  is  projected  to  turn  over  more  rapidly  than  every  18-24  months  and  the 
commercial  supply  chain  is  foreseen  to  become  increasingly  globalized.  As  an  excursion  for 
sensitivity  analysis,  the  workshop  participants  considered  the  case  where  DoD’s  reliance  on 
commercial  products  proves  to  be  a  failure  (e.g.,  commercial  products  fail  to  provide  adequate 
environmental  robustness  or  respond  adequately  to  DoD’s  unique  needs),  prompting  DoD  to  rely 
less  on  commercial  products  and  services. 

4.2.7  Information/Communications.  The  workshop  participants  projected  that  information 
will  be  increasingly  globalized  and  accessible  (via  the  next  generation  internet  and  the  emerging 
global  information  grid).  However,  it  was  observed  that  concerns  will  persist  about  quality  and 
security,  information  assurance  will  remain  elusive,  and  Allied  and  NGO  access  will  remain 
limited  due  to  security  concerns  and  sensitivities.  It  was  further  speculated  that  the 
communications  infrastructure  will  continue  to  put  pressure  on  distributed  M&S,  particularly  for 
nodes  in  an  active  conflict  theater.  Again,  due  to  limited  time  availability,  the  workshop 
participants  opted  not  to  consider  alternative  assumptions  in  this  area. 

4.3  M&S  Technology  Projections.  One  of  the  major  products  of  the  workshops  was  the 
development  and  refinement  of  an  M&S  technology  projection  for  the  next  decade.  This 
consisted  of  three  key  pieces:  a  taxonomy,  an  assessment  of  the  baseline  capability  and  bounding 
ten-year  projections  (i.e.,  conservative  and  aggressive). 

4.3.1  Taxonomy.  As  a  foundation  for  the  technology  projection,  a  taxonomy  was  developed 
that  can  be  depicted  as  a  “jigsaw  puzzle’  with  four  interlinking  pieces  (see  Figure  4). 
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Figure  4.  M&S  Technology  Taxonomy 

These  pieces  include: 

•  Modeling  Methodology  (How  you  model  it).  This  includes  the  theories,  processes, 
algorithms  and  information  that  support  the  conceptualization  of  a  model. 

•  Development  Methodology  (How  you  build  it).  This  includes  the  techniques,  tools  and 
software  used  in  the  design,  constmction,  and  implementation  of  models  or  model  tool 
sets. 

•  Computational  Capability  (How  the  model  executes  and  communicates).  This  includes 
the  systems  on  which  the  M&S  application  will  be  hosted  and  how  the  M&S  application 
connects  to  other  M&S  applications. 

•  Data/Information  Understanding  (What’s  needed  to  build,  transform  and  understand 
it).  This  includes  the  processes  and  tools  for  the  capture,  storage,  transformation  and 
understanding  of  data  and  information. 

4.3.2  Baseline.  As  a  point  of  departure,  the  Technology  Working  Group  used  this  M&S 
technology  taxonomy  to  characterize  the  current  state  of  simulation  technology.  They  arrived  at 
the  following  assessment. 
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•  Modeling  Methodology.  The  most  recent  focus  has  been  on  high  resolution,  entity- 
based,  human-in-the-loop  technologies.  There  has  not  been  comparable  investment  in 
other  key  technology  areas  (e.g..  Computer  Generated  Forces  (CGF);  representation  of 
cognitive  processes;  tools  to  analyze,  capture  and  understand  requirements).  As  a 
consequence,  it  was  concluded  that  simulations  are  still  too  difficult  to  use  and  it  is 
extremely  challenging  to  understand  the  data  that  are  generated  from  simulations. 

•  Design  Methodology.  It  was  recognized  by  the  working  group  that  the  design  challenges 
that  confront  the  simulation  community  are  largely  the  same  as  those  of  the  software 
community.  Consistent  with  that  observation,  it  was  noted  that  the  complexity  of  large 
scale  simulation  systems  is  such  that  they  currently  exceed  the  community’s  ability  to 
specify  and  build  them  effectively  and  efficiently.  In  addition,  standards  are  still 
inadequate  to  support  simulation  interoperability  and  reuse.  Most  of  the  military 
community’s  efforts  are  focused  on  the  High  Level  Architecture  (HLA)  and  Synthetic 
Environment  Data  Representation  and  Interchange  Specification  (SEDRIS). 

•  Computational  Capability.  The  bulk  of  military  simulation  needs  are  not  severely 
constrained  by  existing  computational  constraints.  Tactical  mobile  users  (with  extremely 
limited  communications  bandwidths  and  restricted  computational  resources)  are  an 
exception  to  that  observation. 

•  Data/Information  Technology.  There  are  several  constraining  factors  in  this  area.  First, 
there  are  few  efforts  to  deal  with  the  full  dimensions  of  the  data  problem  (e.g.,  acquire, 
transform,  VV&C  and  warehouse  the  data  needed  by  the  M&S  community).  Second, 
scenario  generation  tends  to  be  a  limiting  factor,  particularly  when  a  large  scale 
simulation  is  being  employed  for  a  theater  that  has  not  been  previously  analyzed.  Third, 
it  is  recognized  that  analysis,  supported  by  simulation,  is  an  art  vice  a  science.  Thus,  the 
education,  skill  and  experience  of  the  analyst  are  driving  factors.  On  the  positive  side, 
collaboration  tools  are  starting  to  mature.  These  tools  are  beginning  to  help  in  the 
requirements  management  process. 

4.3.3  Selected  Technology  Projections.  For  each  of  the  areas  of  the  taxonomy,  technology 
projections  were  made  under  conservative  assumptions  (e.g.,  continuation  of  current  investment 
priorities)  and  aggressive  assumptions  (e.g.,  substantial  increase  in  priority  with  the  subsequent 
likelihood  of  a  breakthrough).  The  complete  set  of  projections  is  included  in  Appendix  B.4.  To 
suggest  the  nature  of  those  assessments,  the  following  section  briefly  summarizes  the  projections 
for  selected  sub-areas  in  each  of  the  taxonomy  categories. 

•  Modeling  Methodology.  One  of  the  key  sub-areas  is  representing  performance 
modulators  (e.g.,  representing  the  effects  of  “soft  factors”  such  as  fear,  morale,  quality  of 
training,  exhaustion).  The  conservative  projection  is  that  the  current  trend  of  ad  hoc 
treatment  of  these  factors  will  continue.  In  that  projection,  models  would  be  highly 
uneven  in  their  treatment  of  such  factors.  Conversely,  an  aggressive  projection  foresaw  a 
state  where  there  would  be  coherent,  community  accepted  models  of  these  factors.  This 
projection  assumed  that  the  SIMTECH  97  recommendation  would  be  ultimately  acted 
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upon  and  an  orchestrated  program  of  “soft  factors”  R&D  would  be  formulated  and 
resourced  appropriately. 

•  Development  Methodology.  There  is  a  trend  towards  employing  composable  technology 
that  will  facilitate  the  development  of  more  “plug  and  play”  M&S.  The  conservative 
trend  is  that  there  will  more  use  of  object  oriented  technology  and  that  component 
technology  will  continue  to  mature.  The  aggressive  assessment  is  that  composable 
technology  will  lead  to  iconic  model  construction.  Such  a  trend  would  lead  to  true  rapid 
prototyping.  Under  those  circumstances,  “boutique”  simulation  would  become  viable 
(i.e.,  a  model  builder  could  rapidly  design  and  implement  a  new  model  to  reflect  the 
specific  issues  of  interest). 

•  Computational  Capability.  Networking  is  a  key  area  that  underlies  the  flexible  use  of 
geographically  distributed  M&S.  The  conservative  projection  is  that  a  10  Gigabit/sec 
backbone  will  emerge  to  support  the  M&S  community  within  the  next  decade  and 
network  setup  will  continue  to  be  challenging.  The  aggressive  projection  is  that  a  very 
broad  bandwidth  backbone  will  emerge  (i.e.,  on  the  order  of  500  Gigabit/sec)  and  that 
network  setup  is  considerably  simplified. 

•  Data  and  Information  Understanding.  One  of  the  key  issues  in  this  area  is  data  capture 
and  storage.  The  conservative  projection  is  that  current  policies  and  processes  persist  for 
a  decade.  Under  those  conditions,  data  capture  and  storage  are  largely  fragmented,  ad 
hoc  and  low  priority.  The  aggressive  projection  assumes  that  the  SIMTECH  97 
recommendation  on  data  is  finally  implemented.  Under  those  circumstances,  one  would 
experience  more  systematic  instrumentation  of  key  events  (e.g.,  advanced  warfighting 
experiments).  Consequently,  database  and  data  warehouse  technologies  would  continue 
to  mature  and  be  used  widely  within  the  M&S  community. 

4.4  Top-Down  Assessments.  Fourth,  top-down  assessments  were  performed  for  the  functional 
areas  of  analysis,  acquisition  and  education  and  training.  These  include  an  articulation  of  a 
vision  for  the  functional  area;  findings,  such  as  an  identification  of  associated  needs  (in  policy, 
management  and  technology)  and  a  characterization  of  perceived  shortfalls  (e.g.,  an  identification 
of  cases  where  technology  needs  exceeded  aggressive  projections  (assessed  as  “red”)  and  cases 
where  technology  needs  fell  between  conservative  and  aggressive  projections  (assessed  as 
“amber”));  a  set  of  recommendations  to  ameliorate  perceived  shortfalls;  and  sensitivity 
assessments  to  establish  the  robustness  of  the  recommendations. 

4.4.1  Analysis  Working  Group. 

Vision.  The  group  defined  a  vision  describing  the  following  operating  circumstances  of  the 
analyst  in  2007.  First,  multidimensional  demands  of  joint,  coalition  and  international  operations 
will  best  be  met  by  conducting  analysis  via  teams  that  mix  the  right  skills  and  experience  to 
answer  pertinent  issues.  Such  teams  will  match  analysts  with  a  broad  range  of  other 
professionals,  including  specially  trained  simulators  and  communicators.  Second,  there  will  be  a 
strong  C4ISR  component  to  analytic  issues,  which  the  simulations  of  the  day  will  be  better 
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prepared  to  address.  The  growth  of  a  new  generation  of  analytic  tools  will  allow  the  analyst  to 
focus  first  on  the  question  of  interest,  settling  on  the  appropriate  tool  for  the  job.  Last,  in  ten 
years,  the  analyst  will  find  it  easy  and  normal  to  work  in  a  distributed  analysis  environment  in 
direct  support  of  the  user,  operator  and  commander. 


Findings.  Subsequently,  priority  needs  were  identified  for  M&S  technology,  procedures,  models 
and  data. 

•  M&S  Technology  Needs.  Table  1  depicts  the  Analysis  Working  Group’s  assessment  of 
its  M&S  technology  needs.  It  can  be  seen  that  the  only  area  in  which  the  group  had 
confidence  that  evolving  technology  would  satisfy  its  needs  is  in  the  area  of 
Computing/Communications. 


Need 

Assessment 

Comments 

. . . mm . . . i . i . . 

mmmm 

liJMMiim  iiujj.iiU.Jii  I . . 

Representation  (Cognitive) 

A/R 

A  limited  capability  exists  only  within  a  restricted 
domain  even  under  aggressive  assumptions. 

Representation  (Physical) 

A/R 

A  capability  exists  only  for  higher  resolution 
models. 

Simulation  Techniques 

A 

Dealing  with  how  entities  organize  themselves 
into  large  groups,  information  flow,  networks, 
aggregation,  etc.  is  a  problem. 

Requirements 

N/A 

Development  Methodology 

System  Composability 

Simulation  Frameworks/Composability 

A 

Standards 

A 

The  rules  for  data  but  not  the  behavior  exist. 

Computing/Communications 

Computing  Platform 

G 

Commercial  advances  will  provide  adequate 

DoD  capability. 

Network  Connectivity 

G 

Commercial  advances  will  generally  provide 
adequate  DoD  capability. 

A 

Security  remains  a  risk  area. 

Data  and  Information  Technology 

Data  Acquisition,  Capture  and  Storage 

A 

Although  means  to  capture  data  are  available, 
there  is  no  clear  process  for  collecting,  storing 
and  sharing  data. 

Transforming  Data 

A 

Aggregation  and  disaggregation  of  data  continue 
to  be  a  major  challenge. 

Presentation,  Understanding  and 

G/A 

Presenting  analysis  results  in  more  than  three 

Collaboration 

dimensions  is  within  the  state  of  the  art. 

However,  communication  of  results  in  terms  that 
the  decision  maker  understands  will  be  a 
problem. 

G:  Green ,  A:  Amber,  R:  Red ,  N/A:  No  assessment  provided  or  not  applicable 

Table  1.  Assessment  of  Analysis  M&S  Technology  Needs 


•  Procedural  Needs.  First,  there  is  a  need  for  procedural  improvements  in  the  area  of 
analytic  design.  In  the  future,  the  analyst  should  have  access  to  a  set  of  tools  that  can  be 
adapted  or  composed  to  suit  the  questions  at  hand.  Second,  there  is  a  need  for  analytic 
procedures  that  allow  detailed  exploration  of  the  study  space.  This  concept  calls  for 
analyses  involving  many  decision  variables  (e.g.,  forces,  systems),  over  a  range  of  several 
scenarios,  with  modifications  to  several  parameters  within  each  scenario.  Third, 
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procedures  are  needed  to  manage  the  data  collection  and  analysis  of  data  from  events  that 
have  some  basis  in  simulation,  are  low  replication  (or  no  replication),  but  are  capable  of 
generating  large  amounts  of  data.  These  procedures  respond  to  the  present  requirement  to 
conduct  “experiments”  or  “demonstrations”  to  support  decision  making  or  program 
development  activities.  This  need  also  involves  a  combination  of  data  analysis 
techniques  and  database  management  tools  to  facilitate  handling  the  data.  Fifth,  there  is  a 
need  is  to  provide  specialized  training  and  continuing  “mentoring”  to  analysts  in  the 
techniques  necessary  to  execute  these  more  complex  analyses.  Finally,  there  is  a  need  to 
continue  development  of  theory  for  new  scientific  approaches  to  military  problems. 

•  Model  Needs.  The  needs  for  models  parallel  those  for  analytic  procedures  and  also 
address  issues  of  expanding  analytic  challenges.  The  highest  priority  need  is  for  a  full 
spectrum  of  tools  that  allow  the  analyst  to  structure  the  analysis  to  fit  the  issue  and  then 
compose  or  select  the  models  and  tools  that  are  appropriate.  The  spectrum  of  tools  must 
include  models  and  tools  for  warfighting  and  non-warfighting  missions.  To  complete  the 
spectrum,  more  specialized  tools  are  required,  including  robust  C4ISR  models,  models 
for  information  warfare,  models  with  non-lethal  effects  and  models  that  address  non¬ 
combat  targets. 

•  Data  Needs.  Many  improvements  are  also  necessary  in  the  area  of  data  support.  First,  it 
is  necessary  to  institute  a  comprehensive  management  process  for  data  management, 
which  includes  identifying  responsible  organizations  for  data  production  and 
management.  Second,  there  is  a  need  for  productivity  tools,  including  scenario 
generation  tools.  The  objective  of  these  tools  is  to  reduce  the  long  number  of  weeks 
required  to  assemble  the  typical  scenario  from  scratch.  Third,  there  is  a  need  for  enabling 
technologies  for  the  extraction  and  analysis  of  useful  data  needs  from  “real  life”  — 
including  warfare,  exercises  and  experiments  like  the  Army  Warfighting  Experiments 
(AWEs)  and  Advanced  Concept  Technology  Demonstrations  (ACTDs).  Fourth,  a  “help 
desk”  is  desired  in  order  to  provide  efficient  access  to  these  improved  capabilities.  Fifth, 
there  is  a  need  for  advanced  mining  tools  to  make  it  possible  to  derive  relationships 
across  databases  or  inferentially  from  data.  Sixth,  the  next  development  desired  is 
intelligent  agents  that  assist  in  generating  specialized  data.  These  agents,  triggered  by 
situations  encountered  in  the  course  of  a  model  run,  would  act  to  gather  detailed  data 
relating  to  cause  and  effect,  providing  greater  clarity  and  insight  for  the  analyst.  Data 
processing  procedures  to  allow  the  specification  of  desired  measures  —  and  then  to 
generate  appropriate  data  —  is  a  further  extension  of  the  desired  data  capabilities. 

Finally,  there  is  a  need  for  a  stronger  ability  to  extract  information  from  data,  including 
visualization  tools  and  processes  for  dealing  with  chaotic  behavior. 

Recommendations. 

•  Education.  Establish  a  formal  educational  course  that  trains  analysts  in  the  techniques 
and  processes  involved  in  complex  analysis.  Moreover,  since  capabilities  will  continue  to 
emerge  and  be  refined,  a  continuing  education  process  is  recommended  to  keep  analysts 
qualified  in  the  latest  techniques. 
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•  Research.  In  order  to  continue  the  advancement  in  analysis,  establish  a  program  of 
continued  research  that  addresses  both  military  phenomenology  and  scientific 
advancement. 

•  Experimentation.  Underpin  AWEs  and  ACTDs  with  an  analytic  process  to  derive  valid 
conclusions  and  usable  data.  This  process  includes  both  a  solid  structure  to  define  the 
experiments  and  analytic  procedures  to  extract  valid  insights  from  the  volumes  of  data 
generated.  To  support  some  of  the  capabilities  already  described,  intelligent  agent 
technology  should  be  pursued. 

•  Dealing  With  Complexity.  Develop  educational  approaches  that  highlight  the  ability  to 
design  a  complex,  highly  dimensional  analysis,  to  execute  it  in  a  distributed  fashion,  and 
to  conduct  a  thorough  analysis  of  the  outputs.  While  the  tools  of  experimental  design, 
stochastic  modeling  and  computer  science  will  fill  much  of  the  need,  education  and 
practice  in  a  focused  curriculum  will  result  in  a  more  responsive,  innovative  analysis. 

•  “New  Sciences.”  Invest  further  effort  in  pursuing  and  developing  the  “new  sciences”  and 
teaching  their  theory  and  application  to  new  practitioners  to  usage  in  appropriate  places. 
The  potential  of  these  “new  sciences”  is  strong,  but  they  need  to  be  focused  on  profitable 
applications. 

4.4.2  Acquisition  Working  Group. 

Vision.  In  the  vision  of  this  group,  acquisition  will  address  total  life  cycle  support,  instead  of 
merely  the  initial  phases  of  concept  development,  design,  development  and  manufacturing. 
Consistent  with  this  focus,  DoD  will  commit  to  life  cycle  cost  as  the  basis  for  making  acquisition 
decisions.  In  addition,  DoD  and  industry  will  use  collaborative,  robust  environments  supported 
by  simulation  technology. 

Some  important  characteristics  are  envisioned  for  the  Simulation  Based  Acquisition  (SBA)  end 
state.  First,  more  of  the  development  responsibility  will  shift  to  industry.  Second,  partnerships 
that  permit  the  sharing  of  data  and  models  will  be  developed  that  have  trust  as  a  cornerstone  of 
the  relationship.  Third,  a  highly  integrated,  electronic  environment  will  be  developed  and 
employed  across  all  life  cycle  functions  to  permit  the  early  and  continuous  visibility  and 
evaluation  of  system  development  alternatives.  Last,  DoD  will  adopt  commercial  practices,  as 
appropriate,  to  reduce  production  times  that  are  much  shorter  than  the  typical  10-15  year  period. 

Findings.  To  implement  that  vision,  major  changes  must  occur  in  culture,  management,  policy 
and  technology/environment.  Cultural  changes,  especially  the  move  to  greater  reliance  on  M&S 
to  reduce  costs,  are  very  important.  There  must  be  integration  across  Services  and  domains.  A 
way  to  foster  trusted  government-industry  partnership  to  promote  pervasive  sharing  of  models 
and  data  must  be  found.  This  last  change  will  not  be  achieved  by  dictate  but  will  necessitate  an 
incentive  for  sharing. 
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The  Acquisition  Working  Group  formulated  the  following  additional  major  findings.  First,  there 
is  a  lack  of  personal  and  programmatic  reward  for  adhering  to  SB  A  tenets.  Second,  there  is 
inadequate  upfront  investment  to  implement  SBA.  Third,  there  is  an  absence  of  investment  in 
the  adoption  or  translation  of  the  best  products  or  lessons  learned  from  other  programs  and  in  the 
underlying  technology,  not  by  individual  programs,  but  rather  by  concerted  programs.  Fourth, 
there  is  a  lack  of  guidance  on  M&S  use  in  formal  acquisition  decisions  and  current  policy  limits 
on  effective  use  (immediate  use  of  current  technology)  of  commercial  domestic  and  international 
products  and  services.  Next,  there  is  a  lack  of  adequate  stewardship  for  M&S  tools,  standards 
and  data  as  well  as  for  M&S  education  and  training.  Last,  there  is  wide  disparity  in  the  extent  to 
which  M&S  technology  will  affect  the  implementation  of  the  SBA  paradigm. 

As  can  be  seen  in  Table  2,  modeling  methodologies  are  the  most  serious  shortfalls  that  require 
high  priority  for  funding. 


Need 

Assessment 

Comments 

Modelinq  Methodology 

Representation  (Cognitive) 

A/R 

Significant  attention  is  required  to  achieve  goals. 

Representation  (Physical) 

A 

Some  qaps  (e.q.,  durability  models)  still  exist. 

Simulation  Techniques 

A/R 

No  serious  efforts  are  underway  to  mature  new 
techniques. 

Requirements 

A 

The  role  of  M&S  in  understanding  system 
requirements  is  unclear. 

Development  Methodology 

System  Composabilitv 

A 

The  software  community  is  drivinq  this  need. 

Simulation  Frameworks/Composability 

G/A 

Standards 

A 

No  standard  for  an  integrated  data  environment 
exists. 

Computing/Communications 

Computinq  Platform 

G 

Network  Connectivity 

G 

Security 

G/A 

Potential  problems  exist  in  protecting  data. 

Data  and  Information  Technology 

Data  Acquisition,  Capture  and  Storage 

G/A 

Basic  tools  exist. 

Transforming  Data 

G/A 

Although  underlying  technology  exists, 
implementations  do  not. _ 

Presentation,  Understanding  and 

G 

Additional  automation  would  be  useful.  Ability  to 

Collaboration 

tie  scenario  goals  to  results  is  not  adequate. 

G;  Green ,  A:  Amber,  R:  Red,  N/A:  No  assessment  provided  or  not  applicable 

Table  2.  Assessment  of  Acquisition  M&S  Technology  Needs 


Recommendations 

•  Program  Objective  Memoranda  (POM)  Investments.  Establish  and  support  sufficient 
M&S-infrastructure  investments  in  the  POM  as  the  norm  to  reduce  life-cycle  costs. 

•  Policy/Guidance.  Establish  policy  and  guidance  on:  M&S  use  throughout  the  entire 
formal  acquisition  decision  process;  utilization  of  emerging  commercial  domestic  and 
international  products  and  services  to  maximize  SBA  potential;  and  guidance  on  a 
common  implementation  for  sharing  M&S  and  data  between  government  and  industry. 
As  part  of  this  policy,  provide  incentives  to  all  the  stakeholders,  including  the  program 
mangers  who  use  these  M&S  methods  properly  for  reducing  life  cycle  costs. 
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•  Institutional.  Establish  a  single  empowered  entity,  not  a  council  or  a  policy 
organization,  to  manage  SBA.  Promote  SBA  partnering  at  many  levels  (e.g.,  the  top  level 
through  M&S  Executive  agents  working  with  industry  conglomerates). 

•  DoD  M&S  Master  Plan.  DMSO  should  identify  model  representations  as  a  priority  in 
the  next  DoD  M&S  Master  Plan.  The  CGF  community  should  reprioritize  and  put  effort 
into  new  simulation  techniques.  DoD  should  work  to  resolve  level  of  abstraction 
difficulties  and  establish  links  between  Computer  Aided  Design/  Computer  Aided 
Manufacturing  (CAD/CAM)  and  operational  effectiveness. 

4.43  Education  and  Training  Working  Group 

Vision.  The  working  group  envisioned  a  future  in  which  individuals  will  be  educated  on  “how  to 
learn.”  Subsequently,  those  individuals  will  receive  training  (i.e.,  “how  to  do”)  that  is  just-in- 
time,  just  enough,  tailored  to  needs  and  delivered  when  and  where  needed.  Consistent  with  that 
vision,  education  and  training  will  be  integrated,  capitalize  on  research  and  leverage  non-DoD 
technology  advances.  In  addition,  analysis,  acquisition  and  education  and  training  will  provide 
mutual  support  and  exploit  common  resources. 

Findings.  First,  education  and  training  will  play  a  more  significant  role  than  ever  before  because 
of  profound  changes  in  the  world.  In  particular,  education  and  training  will  be  needed  to  support 
all  of  the  innovative  Service  programs  (e.g..  Army  After  Next,  Alternative  Air  and  Space  Forces), 
subject  to  the  constraints  of  significant  force  drawdown,  increased  PERSTEMPO  and 
OPTEMPO  and  increased  breadth  of  operations. 

Second,  fewer  resources  for  implementing  education  and  training  will  be  available  because  of  the 
downsizing  trend  across  DoD. 

Third,  as  a  major  paradigm  change,  “just-in-time/just-enough  training,”  distributed  when  and 
where  required,  will  become  an  increasingly  important  and  prevalent  component  of  training.  In 
effect,  the  responsibility  for  learning  will  shift  from  the  institution  to  the  individual  and  will 
require  a  significant  focus  on  teaching  “how  to  learn”  and  thus  an  increased  focus  on  the  science 
of  education.  A  fully  integrated,  constantly  available  education  and  training  system  will  provide 
“pull”  as  well  as  “push”  —  fully  supporting  meeting  the  goals  of  Joint  Vision  2010.  The  goal  is 
to  transition  today’s  classroom  focused  education  process  and  unstructured  hands-on  training  into 
a  continuously  available  distributed  system  composed  of  courses  and  on-line  knowledge  bases 
that  can  be  pulled  by  the  “life-long  student”  as  required,  giving  each  individual  the  resources  to 
become  an  “expert”  when  needed. 

Fourth,  the  training  and  education  process  will  be  significantly  different  because  the  rapid 
evolution  of  technology  will  tend  to  make  detailed,  long-to-develop  training  and  education  plans 
obsolete  before  they  can  be  implemented.  Technology  changes  demand  a  rapid,  flexible  process 
that  can  educate  the  warfighter  quickly  and  satisfy  a  variety  of  technical  capabilities. 
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Fifth,  the  advent  of  new  technologies  will  have  the  potential  to  revolutionize  the  training  and 
education  process.  Distance  Learning  (DL)  and  Advanced  Distributed  Simulation  (ADS)  tools 
will  be  able  to  place  a  user  in  contact  with  geographically  dispersed  knowledge  bases  and 
provide  new  sets  of  simulation  tools  that  may  speed  learning  and  improve  retention.  Web-based 
technologies  such  as  Java  will  help  to  provide  ease  of  application  sharing  across  platforms  that 
will  lead  to  improvements  in  DL  and  ADS.  Advances  in  Artificial  Intelligence  (AI)  hopefully 
will  lead  to  improvements  in  human  behavioral  representation  —  a  key  to  improving  interactive 
training  with  simulations. 

Finally,  M&S  technology  shortfalls  are  broadly  consistent  with  those  identified  by  the  other 
functional  areas  (see  Table  3).  In  general,  the  least  constraining  areas  are  projected  to  be  in  the 
areas  of  Computing/Communications  while  the  most  constraining  areas  are  foreseen  in  Modeling 
Methodology. 


Need 

Assessment 

Comments 

Modeling  Methodology 

Representation  (Cognitive) 

R 

No  viable  HBR  theory  exists. 

Representation  (Physical) 

R 

Societal  effects  (e.g.,  political,  economic  and 
social)  need  to  be  modeled. 

Simulation  Techniques 

R 

No  complete  theory  for  HBR  exists 

No  DoD  Proqram  is  in  place 

Requirements 

A 

DoD  influence  is  necessary 

Development  Methodology 

System  Composability 

A 

Simulation  Frameworks/Composability 

A 

Standards 

A 

The  technology  for  common  infrastructure  and 
standard  interfaces  to  plug  and  play  exist,  but 
operations  is  a  problem. 

Computing/Communications 

Computing  Platform 

G 

Network  Connectivity 

G 

Security 

A 

Data  and  Information  Technology 

Data  Acquisition,  Capture  and  Storage 

A/R 

An  infrastructure  and  mechanisms  to  retrieve 
information  needs  to  be  created.  No  theory 
exists  for  the  collection  of  data. 

Transforming  Data 

G/A 

Presentation,  Understanding  and 
Collaboration 

G/R 

HCI  is  not  standard  across  military  systems.  No 
training  theory  exists. 

G:  Green,  A:  Amber,  R:  Red,  N/A:  No  assessment  provided  or  not  applicable 

Table  3.  Assessment  of  Education  and  Training  M&S  Technology  Needs 


Recommendations.  The  Education  and  Training  Working  Group  formulated  recommendations 
in  six  areas:  training  methods,  needs  assessment,  “come  to  the  people,”  individual  responsibility, 
life-long  process  and  cross-functional  sharing. 

•  Training  Methods.  Develop  new  methods  of  training  in  applying  the  new  technologies. 
DoD  must  adopt  methods  that  will  help  change  the  way  people  learn  in  addition  to  what 
they  learn.  New  learning  methods  that  stress  the  ability  to  assimilate  information  will 
likely  be  required,  instead  of  traditional  methods  that  focused  on  memorization  or 
repetition. 
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•  Needs  Assessment.  Conduct  a  periodic  “Needs  Assessment.”  This  assessment  will:  (1) 
identify  shortfalls  in  the  training  and  education  domains;  (2)  prioritize  these  needs  and 
fund  efforts  to  correct  them  via  an  implementation  plan;  and,  (3)  develop  a  feedback 
process  that  will  periodically  revise  this  implementation  plan. 

•  “Come  to  the  People.”  Make  the  education  and  training  process  significantly  more 
efficient  to  deal  with  the  consequences  of  the  smaller  forces  (downsizing),  the  increased 
OPTEMPO/PERSTEMPO,  and  the  increasingly  complex  world.  This  training/education 
process  must  come  to  the  people,  and  not  the  people  to  it.  It  may  be  prudent  to  oversee 
the  application  of  DL  and  ADS  through  the  formation  of  a  program  office  that  can 
coordinate  the  implementation  across  all  of  DoD  (i.e.,  implementation  of  the  advanced 
distributed  learning  initiative). 

•  Individual  Responsibility.  Individuals  must  take  more  of  the  responsibility  for  training 
and  educating  themselves.  In  support,  DoD  must  adopt  a  policy  that  will  provide 
incentives  for  individuals  to  improve  themselves  through  education  and  training. 

Likewise,  institutions  must  share  in  this  process  so  that  available  resources  are  not 
squandered. 

•  Life-long  Process.  Implement  a  life-long  education  and  training  process  because  the 
world  is  rapidly  changing,  the  rapid  evolution  of  technology  often  makes  knowledge 
obsolete  within  only  a  few  years,  and  each  person  needs  to  be  proficient  in  more  skills 
(fewer  people  engaged  in  more  complex  work).  In  support  of  this  process,  personnel 
systems  must  accommodate  the  need  for  continuous  training  throughout  the  career  cycle. 
To  facilitate  this  process,  broad-based  training  must  be  integrated  with  specific,  tailored 
training  throughout  a  soldier’s  career.  Links  to  non-military  institutions  of  higher 
learning  (e.g.,  universities,  community  colleges)  will  be  necessary  to  expand  the 
knowledge  base  for  such  information.  In  addition  to  the  requirements  for  changes  to  be 
made  in  training  and  education  systems,  numerous  policy  changes  are  needed  to  stimulate 
technology  changes.  Finally,  it  is  important  to  focus  on  how  to  utilize  emerging 
technologies  and  gain  experience  in  applying  them. 

•  Cross-Functional  Sharing.  To  the  extent  possible,  the  Executive  Committee  for 
Modeling  and  Simulation  (EXCMS)  should  focus  on  removing  the  redundancies  in 
applications  of  simulation  across  the  training,  analysis  and  acquisition  areas  of  DoD  and 
leverage  off  similar  efforts.  It  is  recognized  that  some  efforts  in  these  domains  are  unique 
and  cannot  be  leveraged,  but  they  may  be  able  to  utilize  common  core  M&S  pieces. 

In  addition,  management  of  the  analysis,  acquisition  and  training  domains  must  be  done  in  such  a 
way  that  they  may  share  common  resources  when  necessary.  Such  examples  might  include  the 
sharing  of  simulators  (e.g.,  performing  virtual  prototyping  in  the  acquisition  domain,  determining 
requirements  in  the  analysis  domain  and  training  operators  in  the  training  domain).  Data  must  be 
easily  accessible  and  shared  across  domains,  without  the  traditional  concerns  of  releasability  of 
data. 
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5.  Selected  Cross-Cutting  Findings.  Each  of  the  plenary  speakers  at  the  second  workshop 
identified  M&S  as  a  key  enabler  to  promote  revolutions  in  analysis,  acquisition  and  education 
and  training.  This  hypothesis  was  validated  by  all  of  the  working  groups. 

Several  of  the  plenary  speakers  observed  that  many  of  the  obstacles  to  these  revolutions  are 
cultural  in  nature.  Among  the  more  important  cultural  obstacles  identified  were  institutional 
barriers  (e.g.,  the  need  to  go  from  “stovepiped”  organizations  to  more  collaborative  organizations 
that  would  promote  the  more  efficient  sharing  of  tools,  data  and  expertise);  modeling  and 
simulation  barriers  (e.g.,  transitioning  from  the  inflexibility  of  current  M&S  to  more  flexible 
M&S  to  explore  easily  new  operational  concepts,  doctrines,  procedures  and  the  human 
dimension);  and  process  barriers  (e.g.,  transition  from  the  use  of  a  few,  “blessed”  scenarios  to  a 
full  range  of  scenarios  that  span  the  mission  space).  Again,  these  observations  were  extended 
and  validated  by  all  of  the  working  groups. 

From  a  technology  perspective,  the  working  groups  concluded  that  the  most  significant  shortfalls 
were  projected  to  occur  in  modeling  methodology  (i.e.,  adequate  representation  of  key  cognitive 
factors,  performance  modulators,  and  computer  generated  forces);  development  methodology 
(i.e.,  system  architecture/engineering;  system  composability,  scalability;  and  standards  for  design, 
interoperability  and  reuse);  and  data/information  understanding  (i.e.,  tools  for  dealing  with  data 
acquisition,  transformation  and  access;  tools  to  support  collaboration)  [see  Figure  5]. 


Needs  fall  below  Sill  Needs  range  from  ■■  Needs  exceed 

conservative  projections  Just  below  aggressive,  to  aggressive  projections 

Just  above  conservative  projections 


Cognitive 


Performance 

modulators - 

CGF 

Decision  Support 

System/organizational  - 

effects 

Requirements  Process 

BBS! 

-  Scalability 

Composability 

-  System  arch/eng 

Standards 

Sim  Frameworks 
Commercial  games 

Physical 

Human/computer  interface  - 

Analysis 

Collaboration 

Acquire/capture 

Storage 

Accessibility 

Ownership/control 

gyr  _ wp 

Hk  J! 

Network  Speed 
Distributed  computing 

- Network  setup 

Security 

Figure  5.  Aggregate  Comparison  of  User  Needs  and  Technology  Projections 

In  almost  all  cases,  these  projected  technology  shortfalls  cut  across  individual  functional  areas.  It 
is  notable  that  each  functional  working  group  also  opined  that  commercial  developments  in 
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communications  and  computing  would  probably  not  constrain  M&S  applications,  with  the 
exception  of  security  needs. 

6.  Cross-Cutting  Recommendations.  Consistent  with  the  selected  findings,  the  following 
cross-cutting  recommendations  have  been  formulated. 

6.1.  Institutional  Initiatives.  To  facilitate  the  development  of  a  better  balanced  M&S  Science 
and  Technology  (S&T)  investment  strategy,  it  is  necessary  to  develop  a  clear,  comprehensive 
audit  trail  for  current  M&S  S&T  programs  and  plans. 

To  promote  needed  community  sharing  of  tools,  data  and  expertise,  organizational  focal  points 
are  required  for  SBA  and  Advanced  Distributed  Learning  (ADL).  These  organizations  should 
champion  these  processes,  promote  pilot  programs,  monitor  commercial  developments,  begin  to 
establish  the  community  infrastructure  needed  to  “boot  strap”  the  processes  and  assure  the  full 
scope  of  cross-cutting  activities  are  undertaken  (e.g.,  ensure  that  education  and  training  needs  are 
treated  adequately  in  SBA). 

6.2.  Program  Initiatives.  An  expanded  family  of  flexible,  readily  tailorable  M&S  is  needed  to 
address  many  user  needs.  Although  on-going  monolithic  model  developments  (e.g.,  JWARS, 
JSIMS)  will  probably  prove  to  be  central  elements  of  this  family,  they  will  almost  certainly  not 
be  sufficient  to  satisfy  the  needs  of  all  users.  To  complement  them,  “boutique”  models  are 
needed  that  address  all  aspects  of  the  expanding  mission  space  (e.g.,  asymmetric  conflict;  new 
operational  concepts).  These  include  system  dynamics  models  (to  provide  the  ability  to  quickly 
scan  and  pre-filter  scenario  space)  through  virtual  M&S  (to  capture  the  effects  of  distributed 
teams  of  people  under  stress).  In  particular,  the  demands  of  AWEs  mandate  new  classes  of 
M&S,  which  are  sufficiently  flexible  to  explore  easily  new  operational  concepts  and  companion 
education  and  training  experimentation  plans  to  address  the  subjects’  needs  for  associated 
training. 

6.3.  Research  Initiatives.  To  redress  identified  M&S  technology  shortfalls  that  affect  all  users 
of  M&S,  undertake  organized  research  programs  in  “soft  factors”  (e.g.,  cognitive  factors, 
performance  modulators,  computer  generated  forces),  data  (e.g.,  tools  to  capture,  transform,  and 
access  data),  and  selected  subjects  in  fundamental  and  applied  research  (e.g.,  agent-based 
modeling,  search  and  model  building;  variable  structure  simulation;  multi-resolution  modeling; 
role  of  interactiveness  in  discovery  and  analysis).  Mechanisms  must  also  be  established  to 
ensure  that  the  results  of  these  research  programs  are  injected  into  the  practice  of  M&S. 

7.  Summary.  A  great  deal  of  progress  in  military  M&S  has  been  made  over  the  past  decade, 
due  in  no  small  measure  to  the  efforts  of  DMSO.  However,  M&S  is  perceived  to  be  an  enabler 
for  incipient  revolutions  in  analysis,  acquisition  and  education  and  training.  It  will  require 
institutional  initiatives,  an  evolving  family  of  flexible  M&S,  and  a  focused  research  program  to 
realize  those  revolutions. 
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Simulation  Technology  2007 
(SIMTECH  2007):  Overview 


16-18  December  1997  (GRCI) 
18-20  August  1998  (IDA) 


Agenda 


Goals  and  Objectives 
Approach/Participants 
Key  Products 

Overarching  Findings/Recommendations 
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•  To  promote  more  effective  dialogue  between  the 

-  Modeling  and  Simulation  (M&S)  technology  community 

-  Users  of  M&S 

•  analysts 

•  acquirers 

•  educators  and  trainers 


Objectives 


Retrospective 

-  Review  and  assess  the  findings  and  recommendations  that  were  made  by  the 
participants  at  SIMTECH  97 
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Objectives 


Retrospective 

-  Review  and  assess  the  findings  and  recommendations  that  were  made  by  the 
participants  at  SIMTECH  97 

Looking  forward 

-  Identify  and  prioritize  the  needs  of  the  users  of  military  M&S 

-  Assess  the  probable  evolution  of  M&S  technology  over  the  next  decade 

-  Identify  potential  user  shortfalls  (e.g.,  policy,  management,  technology)  and 
recommend  actions  to  ameliorate  them 


Agenda 


•  Goals  and  Objectives 


Key  Products 

Overarching  Findings/Recommendations 
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Workshop  Organizers 


Position 

Name 

Affiliation 

Chair 

Stuart  Starr 

MITRE 

Co-chairs 

Howard  Carpenter 

MITRE 

Denis  Clements 

GRCI 

Lana  McGlynn 

DAMO,  USA 

Bob  Orlov 

J-8,  JS 

Bob  Statz 

BAH 

MORS  “Bulldog” 

Col  “Crash”  Konwin 

DMSO 

Plenary  Speakers 


Speakers  Workshop  I 


Keynote  Ed  Brady,  SPI 


Plenary  Lou  Finch,  OSD 
Eric  Coulter,  OSD 
Robin  Frost,  OSD 

I  Paul  Davis,  RAND 


Workshop  II 


Larry  Welch,  IDA 


VADM  Pat  Tracey,  USN 


Vince  Roske,  Joint  Staff 


Pat  Sanders,  OSD 


Vem  Bettencourt,  FS,  AMSO,  USA 


Gary  Yerace,  DMSO 


Paul  Davis,  RAND 


Dick  Nance,  VPI 


Richard  Weatherly,  MITRE 


Steve  Bankes,  RAND 
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Working  Group  Leadership 


Theme 

Leadership 

Life  Cycle  Management 

Bruce  Bennett,  RAND 

Bob  State,  BAH 

Analyst  Workbench 

Iris  Kameny,  RAND 

John  Gilmer,  Wilkes  University 

“Soft”  Factors 

Dick  Hayes,  EBR 

Lasbon  Booker,  MITRE 

Data 

Phil  Dickinson,  Consultant 

Cart  Russell.  JNTF 

Analysis 

Cy  Staniec,  Logicon 

Hank  Dubin,  OPTEC 

Russ  Richards,  MITRE 

Mark  Youngren,  MITRE 

Acquisition 

Gary  Jones,  DARPA 

Dick  Helmuth,  SAIC 

Annie  Patenaude,  SAIC 

Dell  Lunceford,  DARPA 

Education  and 

Training 

Phil  A  bold,  A&B  Technology 

Zach  Furness,  MITRE 

Julia  Lough  ran.  Thought!  Ink 

Technology  Assessments 

Dell  Lunceford,  DARPA 

Denis  Clements,  GRCI 

Julia  Lough  ran.  Thought!  ink 

Synthesis 

Paul  Davis,  RAND 

Clavton  Thomas,  FS,  USAF 

Workshop  Participants 


Workshop  I 


Workshop  II 


Government 

(Non-Military 

Government 

(Military) 


0)^M=6 


Academia 


jW  International 


Total  =  83 
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Agenda 


•  Goals  and  objectives 

•  Approach/participants 

•  Key  products 

-  Lessons  learned  from  SIMTECH  97 

-  Key  drivers/assumptions 

-  Technology  projections 

-  Functional  assessments 

•  Overarching  findings/recommendations 


What  Did  SIMTECH  97  Do  Well/Less  Well? 

*  Did  well 

-  Helped  set  the  Science  and  Technology  (S&T)  agenda  for  DMSO,  focus  Service 
investments 

-  Helped  shape  community  perspectives  on  M&S  (e.g.,  view  lifecycle  management 
as  an  evolving  process) 

•  Did  less  well 

-  Made  an  implicit  assumption  of  constancy,  without  adequate  sensitivity  exploration 

-  Failed  to  anticipate  key  changes;  e.g., 

*  Geopolitical  (e.g.,  end  of  Cold  War) 

•  Technological  (e.g.,  exponential  growth  of  Internet;  DoD-commercial  software  spin- 
off/spin-on  process) 

-  Gave  excessive  emphasis  to  subordinate  issues  (e.g.,  massively  parallel  systems) 
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Selected  Recommendations 
(Based  on  Retrospective  Look) 


Document  key  assumptions  more  explicitly 
Perform  sensitivity  explorations 

Don’t  generate  “point”  solutions  (i.e.,  it  is  better  to  be  approximately  right 
than  precisely  wrong) 

Work  assiduously  to  disseminate  the  workshop  findings, 
recommendations;  e.g., 

-  Identify,  work  with  a  “champion”  (e.g.,  DMSO) 

-  Spread  the  word  beyond  the  ‘"usual  (MORS)  suspects” 

Continue  to  push  for  action  on  key  residual  recommendations;  e.g., 

-  Undertake  “Soft  Factors”  research  program 

-  Systematically  collect  and  “transduce”  data  from  key  targets  of  opportunity  (e.g., 
AWEs,  ACTDs) 


Macro  Environmental  Assumptions  for  2007 


fn w=a 
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Selected  Sensitivity  Excursions 


Mission  trends 

-  A  peer  competitor  emerges  prior  to  2007 

Strategy/operational  trends 

-  Joint  Vision  201 0  is  perceived  as  unsuccessful  (i.e.,  return  to  more  deliberate/mass 
oriented  operations) 

Institutional  trends 

-  “Top  line”  defense  budget:  +/-20%  from  the  baseline 

Systems  trends 

-  Limited  acquisition  resources  prompts  the  DoD  to  plan  to  retain  systems  for  50% 
longer  than  existing  systems 

-  DoD  shifts  to  an  acquisition  strategy  that  emphasizes  more,  cheaper  (possibly 
lower  tech)  systems 

Commercial  trends 

-  Reliance  on  commercial  products  proves  to  be  a  failure,  prompting  DoD  to  rely  less 
on  commercial  products,  services 
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Technology  WG:  Approach 


•  Taxonomy 

-  Several  taxonomies  for  structuring  the  M&S  technology  assessment  problem  were 
formulated  and  applied 

-  Ultimately,  it  was  elected  to  employ  a  taxonomy  structured  around  the  concept  of  a 
“jigsaw  puzzle”  (as  seen  below) 

•  Assessments 

-  Within  the  context  of  the  taxonomy,  two  technology  assessments  were  performed 
for  the  year  2007 

•  Conservative  predictions  assume  that  the  area  maintains  its  current  priority  and  funding  in 
the  area  is  consistent  with  currently  programmed  resources 

•  Aggressive  predictions  assume  that  die  area  receives  a  substantially  higher  priority  and 
funding  in  the  area  is  increased  accordingly 
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M&S  Technology  Taxonomy 

*  ij 

Modeling 

Development 

Methodology: 

Methodology: 

How  you  model  it 

How  you  build  it 

Data/Information 

Computational 

Understanding: 

Capability: 

What’s  needed  to 

How  the  model 

build,  transform,  and 

executes  and 

understand  it 

communicates 

m»a 
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Technology  Areas 


•  Modeling  methodology:  The  theories,  processes,  algorithms  and 
information  that  support  the  conceptualization  of  a  model 

•  Development  methodology:  The  techniques,  tools  and  software  used  in  the 
design,  construction  and  implementation  of  the  models  or  the  model  tool 
set 

•  Computational/communications  capability: 

-  What  the  M&S  application  will  be  hosted  on 

-  How  it  connects  to  other  M&S  applications 

-  How  M&S  application  developers  and  users  connect  to  one  another 

•  Data/information  understanding:  The  tools  and  processes  for  data 
capture,  storage,  transformation  and  understanding: 

-  How  to  acquire  and  store  the  data 

-  How  to  transform  it  (e.g.,  scenario  generation) 

-  How  to  understand  the  data  (including  HCI  factors  and  information  understanding 
techniques) 

-  How  to  collaborate  about  it 
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Current  State  of  Simulation  Technology 


•  Modeling  methodology 

-  Most  recent  focus  on  high  resolution,  entity  based,  human-in-the-loop  technologies; 
little  real  investment  in  other  areas 

-  Few  to  no  advances  in  Computer  Generated  Force  (CGF)  technology 

-  Simulations  still  too  hard  to  use,  too  hard  to  understand  results 

•  Design  methodology 

-  Simulation  =  software;  many  of  the  development  challenges  of  one  are  the 
challenges  of  the  other 

-  Large  scale  simulation  system  complexity  exceeds  community’ s  ability  to  build 

-  Standards  efforts  currently  (esp.  for  interoperability/reuse)  limited  to  HLA  and 
SEDRIS;  most  standards  in  use  ‘owned’  by  commercial  sector 

•  Computational  capability 

-  A  limiting  factor,  but  not  by  much  for  bulk  of  simulation  needs 

•  Data/information  technology 

-  Few  efforts  to  instrument/collect  raw  data 

-  Scenario  generation  a  limiting  factor 

-  Analysis  an  art,  not  a  science 

-  Collaboration  tools  maturing; 
beginning  to  help  requirements  management 
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Selected  Technology  Projections 


Area 

Sub-Area 

Conservative 

Aggressive 

Modeling 

Methodology 

Representing 

Performance 

Modulators 

•  Approaches 
remain  ad  hoc 

•  Coherent,  community- 
accepted  models* 

Development 

Methodology 

Composable 

Technology 

•  More  use  of  Object 
Oriented  Technology 

*  Maturing  of  component 
technology 

•  Development  leads  to  iconic 
model  construction 

•  True  rapid 

prototyping/”  boutique” 
simulation  becomes  viable 

Computation  and 
Comms 

Network 

•  10  Gigabit  backbone 

•  Network  setup  continues 
to  be  difficult 

•  500  Gigabit  backbone 

*  Network  setup  is 
considerably  simplified 

Data  and  Info 
Understanding 

Data  capture, 
Storage 

•  Continues  to  be 
—Fragmented 
-Ad  Hoc 
—Low  priority 

•  More  systematic 
instrumentation  of  events* 

•  Database,  data  warehouse 
technologies  mature 

*  Assumes  SIMTECH  97  recommendations  are  implemented 
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Guidance  to  the 
Functional  Working  Groups 


•  Formulate  your  vision  for  the  future 

(on  how  M&S  can  enhance  effectiveness,  efficiency) 

•  Identify  the  key  drivers  that  influence  your  needs 

•  Identify  and  prioritize  needs  to  achieve  the  vision  in  the  areas  of 

-  Policy 

-  Management 

-  Technology 

-  Other 

•  Provide  recommendations  to  address  high  priority  needs 

•  Perform  sensitivity  analyses  to  assess  the  robustness  of  your 
recommendations 
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Education  and  Training  Vision 


•  What 

-  Educate  on  “how  to  learn” 

-  Train  (“how  to  do”) 

•  just-in-time 

•  just  enough 

•  tailored  to  needs 

•  delivered  when,  where  needed 

•  How 

-  Education  and  Training  will 

•  be  integrated 

•  capitalize  on  research 

•  leverage  non-DoD  technology  advances 

-  Analysis-acquisition-education/training  will 

•  provide  mutual  support 

•  exploit  common  resources 


Key  Drivers:  Education  and  Training  (E&T) 


Mission  Trends 

•  Expanded  mission  space 

•  Changed  ACOM  mission 


E&T  M&S 
Needs 


Strategy/Operational  Trends 

•  Increasing 

-  Operations  space 

-  Decision  space 

•  Reduced  decision  time 

•  Increased  reach  back  operations 


Training  Trends 

•  Environmental  changes  imply  new  E&T 

-  Requirements  (e.g.,  need  for  lifelong 
learning/training;  ability  to  train  with 
others) 

-  Constraints  (e.g.,  limited  time  to  train) 
.  New  technologies  pose  new 

opportunities 

-  Advanced  Distributed  Learning  (ADL) 

-  Continuous  training  opportunities 


Institutional  Trends 

•  Downsizing  of  force 

•  Merging  of  classical  stovepipes 

-  Operations,  training 

-  M&S,  C2 
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Key  E&T  Technology  Needs 


Assessment 


Modeling 

Methodology 


Development 

Methodology 


Data  and 

Information 

Technology 


*Lcgcad:  Symbol 

Red  (R) 
Amber  (A) 


•  Improve  behavioral  representation _ 


•  Broaden  representation  of  societal  effects 
(e.g.f  political,  social,  economic) _ 


•  Adapt  to  personal  learning  style/distributed 
learning 


•  Create/adopt  open  architectures  which 
allow  integration  of  COTS/GOTS  products 


•  Create  a  common  infrastructure  and 
standard  interface  to  support  Uplug  and 
play” 


•  Address  security  issues 


•  Data/information  acquisition  (including 
operator,  system  instrumentation) 


•  Standardization  of  HCI  across  military 
systems 


Mmafa* 

Needs  exceed  aggressive  projection 
Needs  range  from 

-  Just  below  aggressive  to 

-  Just  above  conservative 
Needs  fall  below  conservative  projection 


Key  Education  and  Training  Recommendations" 


Institutional:  Incorporate  E&T  into: 

-  Simulation  Based  Acquisition 

-  Joint  Experimentation  Program 

Research:  DDR&E  should  establish  a  research  program  to 

-  Capture  and  extend  theory  on 

•  “how  we  learn” 

•  “how  to  teach” 

-  Develop  human  performance  measures/metrics  to  support  E&T  evaluation 

-  Capture,  store,  and  make  accessible  information  on 

•  individual  and  organizational  performance 

•  E&T  system  performance 

-  Create  a  comprehensive  program  on  human  behavior  representation 
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Analysis  Vision 


What 

-  Analysts  will  have  the  ability  to  address  a  broad  range  of  problems  (e.g., 
joint/interagency /coalition),  with  timely,  accurate  and  relevant  insight  and 
information 

How:  Analysts  will  work 

-  In  distributed,  interdisciplinary  teams;  e.g.,  analysts,  M&S  experts,  C4ISR  experts 

-  In  support  of  commanders  and  decision  makers  at  all  levels  (wherever  they  are) 

-  Either  directly  or  through  powerful  new  analyst-developed 

•  tools 

•  decision  aids 

•  data  bases 


Appendix  A  -  15 


Analysis  Recommendations  (1  of  2) 


*  Tool  chest 

-  Create,  sustain  M&S  that  treat  C41SR  explicitly  to  deal  with  all  operations  of 
interest,  particularly 

•  OOTW 

•  New  concepts  of  warfare;  e.g., 

-  Split  base  operations 

-  Network  centric  warfare 

-  Information  warfare 

-  Small  unit  operations 

-  Non-lethal  weapons 

-  Augment  existing  tools  with  the  latest 

•  Conceptual  thinking  (e.g.,  complexity  theory) 

•  Techniques  (e.g.,  agent  modeling,  genetic  algorithms) 

-  Incorporate  new  features ;  e.g., 

•  Intelligent  agents  to  “bookmark”  key  events  to  facilitate  “what  if. . .  ”  analyses 

•  Better  visualization  capabilities 

•  Data  mining,  knowledge  discovery  tools 


Analysis  Recommendations  (2  of  2) 

•  Procedures 

-  Create  the  ability  to  design  analyses  to  suit  issues 

-  Ensure  that  analysts  explore  “study  space”  fully 

-  Transform  “demonstrations”  into  “experiments”  (e.g.,  AWEs) 

-  Enhance  capability  to  analyze  large  volumes  of  complex  data  effectively 

-  Undertake  a  research  program  to  develop  new  scientific  approaches  to  military 
problems 

•  Data 

-  Institute  a  comprehensive  process  for  data  management 

-  Develop  technologies  for  extraction  and  analysis  of: 

•  useful  data  from  events  (e.g.,  intelligent  agents) 

•  information  from  data  (e.g.,  visualization  tools) 

-  Establish  a  “Help  Desk”  for  data 

•  People 

—  Educate/train/mentor  analysts  on  the  the  new  procedures,  tools,  data 
management/technologies 
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Acquisition  Vision 


What 

-  Reduce 

•  time 

•  resources 

•  risk 

•  total  ownership  costs  throughout  life  cycle  process 

-  Increase 

•  quality 

•  military  worth 

•  supportability  of  fielded  systems 

How 

-  Enabled  by  simulation  technology  that  is 

•  Robust 

•  Used  collaboratively 

•  Integrated  across  life  cycle  phases,  functions  and  programs 


Acquisition  Recommendations  (1  of  2) 


Policy 

-  Establish  policy/guidance  on  the  use  of  M&S  in  formal  acquisition  decisions 

-  Define  a  policy  to  utilize  emerging  commercial  domestic  and  international 
products,  services  to  maximize  Simulation  Based  Acquisition  (SBA)  potential 

-  Establish  DoD  policy  and  a  common  implementation  for  sharing  M&S  and  data 

Organizational 

-  Identify  and  empower  an  organization  to  enable 

•  dedicated,  enduring  pilot  and  flagship  programs 

•  stewardship  for  managing  SBA 
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Acquisition  Recommendations  (2  of  2) 


•  Institutional 

-  Establish  and  support  sufficient  M&S  infrastructure  investments  in  the  POM 

-  Provide  incentives  to  all  stakeholders,  accompanied  by  adequate  up  front 
investment,  for  use  of  M&S  early  and  throughout  the  life  cycle 

•  Technology 

-  Ensure  that  the  next  DoD  M&S  Master  Plan  highlights  key  SBA  needs 

•  improved  representation  of  emerging  warfare  areas,  cognitive  factors 

•  enhanced  CGF 

•  standards  for  design,  interoperability  (i.e.,  links  between  CAD/CAM  and  operational 
effectiveness  M&S  to  support  “what  if...”  trades) 


Agenda 

•  Background 

•  Goals  and  Objectives 

•  Approach 

•  Key  Products 
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Cross-Cutting  Findings  (1  of  2) 


M&S  is  perceived  to  be  a  key  enabler  to  promote  revolutions  in 

-  Analysis 

-  Acquisition 

-  Education  and  training 

Many  of  the  barriers  to  these  revolutions  are  cultural  in  nature 

-  Institutional  (e.g.,  need  to  move  from  “stovepipes”  to  broader  collaboration) 

-  M&S  (e.g.,  transition  from  Cold  War  models  to  those  that  reflect  the  New  World 
Disorder) 

-  Process  (e.g.t  move  from  a  few  “blessed”  scenarios  to  broad  explorations  of  the 
“scenario  space”) 


Some  Needed  Cultural  Changes  (re:  GEN  Welch) 

(1  of  2) 


Area 

Sub-Area 

From 

To 

Institutional 

Outcomes 

Protected 

Open 

Review 

Bureaucratic 

Subject  matter  expert 

Organizational 

Stovepiped 

Collaborative 

M&S 

Orientation 

Model 

Subject  matter 

Processes 

Opaque 

Transparent 

Data 

Limited  to  “validated” 

Include  possibilities 

Algorithms 

Stable,  traceable 

Allow  for  non-linear 

Mission  Orientation 

Cold  War 

New  World  Disorder 
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Some  Needed  Cultural  Changes  (re:  GEN  Welch) 

(2  of  2) 


Area 

Sub-Area 

From 

To 

Processes 

Uncertainty 

Suppress 

Illuminate 

Scenarios 

Few,  “blessed” 

Full  range 

Analysis 

Force-on-force 

Military-social-economic 

Forces,  concepts 

Symmetrical 

Asymmetrical 

Trade-offs 

Limited 

Full  range 

Scope 

Narrow 

(Force  structure; 
Military  worth  of 
equipment) 

Broad 

(Forces;  equipment; 
doctrine;  concepts; 
tactics;  C2; ...) 

Cross-Cutting  Findings  (2  of  2) 


The  findings  of  the  Functional  Working  Groups  were  generally  insensitive 
to  changes  in  selected  assumptions 

From  a  technology  perspective,  there  was  a  broad  consensus  that 

-  Commercial  developments  in  communications/computing  would  probably  not 
constrain  M&S  applications  (except  for  security  needs) 

-  Significant  shortfalls  were  projected  to  occur  in 

•  modeling  methodology 

•  development  methodology 

•  data/information  understanding 

•  [See  next  vu graph  for  details] 
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Summary 


These  findings  and  recommendations  should  provide  a  sound  foundation 
for  coordinated  community  actions  for  the  foreseeable  future 
SIMTECH  2007  met  the  needs  of  the  community  in  several  dimensions;  it 

-  Articulated  visions  of  how  analysis,  acquisition,  and  education  and  training  might 
be  performed  in  2007 

-  Characterized  projected  conservative  and  aggressive  bounds  on  M&S  technology  in 
2007 

-  Identified  areas  where  projected  M&S  technology  will  not  satisfy  the  needs  of  key 
functions 

-  Recommended  initiatives  to  redress  shortfalls 
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Report  of  the 
Analysis  Working  Group 


The  report  documents  the  deliberations  of  the  Analysis  Working  Group.  The  group 
established  a  vision  of  the  operational  environment  as  it  may  exist  in  ten  years,  then  enumerated  a 
set  of  needs  determined  by  the  operational  environment.  The  needs  were  coupled  with  trends 
projected  by  the  technology  group,  resulting  in  assessments  of  future  capabilities.  The 
assessments  and  resulting  recommendations  are  documented  in  this  report. 

The  workshop  was  held  in  two  parts,  in  December  1997  and  August  1998. 
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The  working  group  was  chaired  by  Dr  Cyrus  Staniec.  Part  1  was  co-chaired  by  Dr  Hank 
Dubin  and  Dr  Russ  Richards.  Part  2  was  co-chaired  by  Denis  Clements  and  Dr  Mark  Youngren. 
Special  editorial  help  was  provided  by  Robert  Statz. 

Both  groups  was  comprised  of  a  number  of  senior  military  analysts  representing  a  vast 
number  of  years  of  analytic  and  simulation  experience.  The  participants  listed  on  this  slide 
represent  direct  experience  with  the  Army,  Navy  and  Air  Force.  Skills  included  test  and 
evaluation,  analysis  and  model  development,  as  well  as  a  blend  of  experience  from  theater  warfare 
analysis  to  low  intensity  conflict  and  operations  other  than  war.  Accordingly,  the  group  was  well 
suited  to  undertake  the  assigned  mission. 
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Military  Operational  Environment 
(Analysis  driver/consumer) 

•  JV2010:  Speed  and  synchronization  of  effects 

-  Reduced  participant  density  per  unit  measure  of  lethality 

•  Joint/coalition/intergovernmental/NGOs 

•  Information  Warfare  (IW)  and  dominance 

-  Information  quality/reliability 

-  Real-time  information  demand 

-  Information  overload 

•  Information  proliferation:  International/organizational 

-  Policy  influences 

•  Scenario/Mission:  MTW/OOTW/urban 

-  US  military  as  support  to  main  mission 

-  Non-lethal  objectives 

•  Threat  concerns:  Asymmetric  threats/RMA/WMD 

•  Resource  constraints  (competition  with  entitlements  demands) 

•  Use  of  AWEs,  ATDs,  ACTDs,  exercises,  SBA  to  support  decision  making 


In  order  to  forecast  the  simulation  needs  of  military  analysts,  it  is  important  to  first  consider 
the  military  operational  environment  that  will  exist  ten  years  in  the  future.  Although  this  is  in 
itself  risky,  there  are  certain  trends  and  intents  of  the  Department  of  Defense  (DoD)  that  improve 
the  odds  of  making  correct  forecasts. 

First,  it  is  the  clear  intent  of  the  Department  of  Defense  (indeed,  even  Congress)  to  implement 
Joint  Vision  2010.  The  implication  of  this  decision  is  that  analytic  support  must  be  provided  to 
forces  that  will  strive  for  information  dominance  and  execute  operations  through  speed  and 
synchronization  of  effects.  Analytic  simulation  must  support  needs  related  to  the  JV2010  pillars: 
precision  engagement,  dominant  maneuver,  full  dimensional  protection  and  focused  logistics. 

Furthermore,  it  seems  likely  that  current  operational  trends  will  continue,  especially  if 
funding  resources  remain  tight  for  the  next  several  years.  Whether  or  not  there  is  a  theater  war, 
operational  tempo  will  likely  remain  high.  Joint,  interagency,  coalition  and  international  agency 
operations  will  still  influence  military  operations. 

The  growth  of  information  will  continue  as  we  seek  information  dominance,  and  we  will  be 
faced  with  the  need  to  preclude  fatigue  due  to  proliferation  of  information. 

Sustaining  our  lead  in  conventional  warfare,  our  focus  will  remain  on  asymmetric  threats  and 
operations  other  than  war  as  drivers  of  DoD  future  developments. 
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Vision 


•  Analysts  supporting  commanders  and  decision  makers  at  all  levels  with 
timely,  accurate  and  relevant  insight  and  information,  either  directly  or 
through  powerful  new  analyst-developed  tools,  decision  aids  and  data 
bases. 
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As  a  result  of  the  anticipated  operational  environment  outlined  on  the  previous  slide,  the 
vision  on  this  slide  describes  the  operating  circumstances  of  the  analyst  in  2007.  The  group 
viewed  decision  making  in  two  broad  categories:  operational,  time-dependent,  decision  making 
and  deliberate,  less  time  sensitive  decision  making. 

In  the  first  case,  the  vision  suggests  analysis  support  “at  the  commander’s  elbow,”  either 
through  responsive  analysis  “on  the  fly,”  or  through  decision  tools,  data  displays  or  other  aids  that 
provide  real  time  input  to  the  commander’s  decision  process. 

In  the  second  case,  the  vision  suggests  analysis  using  advanced  methods  and  in  more  detail  to 
provide  insight  into  difficult  decision  alternatives.  Issues  may  include  “system  of  system”  designs 
or  supporting  analysis  to  the  Quadrennial  Defense  Review  (QDR). 

In  either  case,  there  is  a  flavor  of  faster,  more  timely,  and  better  in  the  demand  for  decision 
information. 
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Analytic  Environment  (1  of  2) 


•  Analysts  will  increasingly  work  in  cooperative,  distributed 
multidisciplinary  teams  to  rapidly  and  scientifically  address  a  broad 
range  of  problems  considering  many  diverse  factors  involving  high  levels 
of  uncertainty. 

•  Analysts  must  understand  the  three  major  players  (users,  modelers  and 
analysts)  and  their  roles. 

•  Time  critical  (COA  Analysis)  vs.  non  time  critical  (resource  allocation) 

•  Analysis  of  the  issue,  not  focus  on  the  tools 

•  Different  approaches  will  apply: 

-  M&S  for  Insight,  Vs. 

-  “Grand  style”  modeling 

•  High  Fidelity 

•  V&V  required 

•  More  customer  involvement,  particularly  in  COA  analysis 
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The  analytic  environment  will  be  characterized  by  the  multidimensional  demands  of  joint, 
coalition  and  international  operation.  These  demands  will  best  be  met  by  conducting  analysis  via 
teams  that  mix  the  right  skills  and  experience  to  answer  pertinent  issues.  Such  teams  will  match 
analysts  with  a  broad  range  of  other  professions,  including  specially  trained  simulators  and 
communicators. 

The  competing  demands  of  the  analyst’s  customers  and  those  providing  technical  services, 
like  modeling,  will  be  similar  to  today.  The  technology  will  permit  faster  and  better  time-critical 
analyses  to  facilitate  Course  of  Action  analyses  in  the  field.  The  analyst  should  not  let  the 
technological  cornucopia  interfere  with  the  use  of  good  analytic  approaches.  The  analyst  must 
focus  on  the  issues,  not  the  tools. 

Quick  response  tool  technology  will  continue  to  evolve,  permitting  increasingly  sophisticated 
analysis.  They  will  be  used  to  provide  high  order  insights,  and  therefore  will  be  less  likely  to 
require  extensive  V&V  as  large  scale,  high  fidelity  models. 

There  will  continue  to  be  a  requirement  for  customer  involvement,  particularly  in  critical 
analyses,  and  the  technology  will  facilitate  such  collaboration. 


Appendix  B.l  -  5 


Analytic  Environment  (2  of  2) 

•  Organizing  for  Analysis 

-  Joint/coalition 

-  Intergovemmental/intemational/interagency 

-  Distributed 

-  Collaborative 

-  Interdisciplinary  teams  to  handle  new  concepts  (e.g.,  flow  of  refugees) 

•  Operator-analyst  linkage 

•  Modelei/Data  specialist 

•  Model  as  enabler  of  interdisciplinary  approach  (visualization) 

•  Outsource  (erosion  of  DoD  internal  capability) 

•  New  MOEs  replace  attrition  as  major  discriminator  between  alternatives 

•  Uncertainty  in  model  design 

-  May  not  know  what  all  the  critical  factors  are 


It  is  likely  that  there  will  be  increased  collaboration  in  a  distributed  analytic  environment. 
Interdisciplinary  teams  will  be  required  because  of  the  need  to  examine  concepts,  such  as  the  flow 
of  refugees,  that  in  the  past  were  too  large,  too  difficult  to  understand,  or  too  far  removed  from  the 
central  focus  of  the  modeling  tools  to  be  properly  examined.  Both  the  strategic  focus  of  the 
military  on  OOTW  and  the  availability  of  new  tools  will  allow  better  examination  of  such  issues. 

The  analyst  will  be  challenged  by  this  same  environment  to  provide  more  focused  Measures 
of  Effectiveness  (MOEs)  as  problems  are  examined  that  differ  significantly  from  the  traditional 
large  scale  force  analyses.  This  issue  is  discussed  in  more  detail  later. 

The  analyst  will  continue  to  be  challenged  by  the  fact  that  models  developed  by  others  will  be 
largely  opaque,  not  because  of  modeling  and  computational  technology,  but  because  the 
understanding  of  human  factors  will  lag  the  ability  of  machines  to  simulate  it.  Many  critical 
human  factors  will  not  be  identified,  let  alone  understood  in  the  next  ten  years. 
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Prioritized  Needs  for  Procedures  (1  of  2) 


•  Ability  to  design  the  analysis  to  suit  the  issue 

-  Then  adapt/compose  the  tools  necessary  to  address  the  questions 

-  Able  to  respond  with  either  quickness  or  depth 

-  Anticipate  new  scenarios,  doctrine,  MOE 

-  Easy  development  of  robust  alternatives 

•  Capability  to  explore: 

-  The  “study  space”  to 

•  (Scenario  space,  sensitivity  space  and  multidimensional  structural  variables) 

-  Evaluate  altematives/courses  of  action,  to  identify  resource  implications,  robust 
courses  of  action 

-  Employing 

•  Parallel  runs,  data  collection  federation 

•  Visualization  tools 

•  Principle  components  analysis 

•  RAND  multidimensional  visualization  tool 
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The  first  priority  need  for  procedural  improvements  is  in  the  area  of  analytic  design.  Current 
capabilities  require  a  lot  of  time,  effort  and  modification  to  existing  tools  and  models  to  conduct 
demanding  analyses.  The  alternative  is  to  conduct  analyses  with  ill-suited  tools,  accepting 
questionable  fits  or  questionable  results. 

In  the  future,  the  analyst  should  have  access  to  a  set  of  tools  that  can  be  adapted  or  composed 
to  suit  the  questions  at  hand.  These  tools  may  be  constructive  models,  or  other  simulations 
coupled  with  advanced  techniques  for  conducting  analysis.  Considering  the  range  of  possible 
analytic  applications,  this  capability  must  be  coupled  with  the  ability  to  adjust  resolution  and  run 
speed  to  needs.  It  must  be  possible  to  select  low  resolution  to  do  quick-turn  analysis  for  one 
application,  or  to  select  higher  resolution  for  a  more  accurate,  detailed  analysis. 

The  second  priority  need  is  for  analytic  procedures  that  allow  detailed  exploration  of  the 
study  space.  This  concept  involves  being  able  to  conduct  an  analysis  involving  many  decision 
variables  (e.g.,  forces,  systems),  over  a  range  of  several  scenarios,  with  modifications  to  several 
parameters  within  each  scenario.  Features  of  this  capability  include  the  ability  to  process  runs  in 
parallel  to  achieve  volume,  along  with  convenient  data  management  and  collection  abilities  in 
model  federations.  Advanced  visualizations  are  also  required  to  assist  in  analyzing  and  presenting 
multidimensional  information.  One  such  approach  to  visualization  is  the  method  developed  by 
RAND. 
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Prioritized  Needs  for  Procedures  (2  of  2) 

•  Analytic  approaches  to  A  WEs,  ATDs,  ACTDs,  exercises 

•  Better  capabilities  to  “explain”  what  was  observed;  traceability;  audit 
trail 

•  Training/mentoring:  How  to  do  analysis  in  a  complex  environment 

•  Research:  Develop  analytic  applications  for  “new  science” 

-  Complexity  theory,  genetic  algorithms,  agent-based  modeling 

-  Data  mining  (intelligent  data  retrieval) 

-  Decision  support  objects  (post  processors) 

•  Joint  experimentation  to  provide  analytical  constructs 
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The  third  priority  responds  to  the  present  tendency  to  conduct  “experiments”  or 
“demonstrations”  to  support  decision  making  or  program  development  activities.  Most  of  these 
events  have  some  basis  in  simulation,  are  low-replication  (or  no  replication)  events,  but  are  capable 
of  generating  large  amounts  of  data.  The  challenge  for  the  future,  therefore,  is  to  develop 
procedures  for  these  events  to  manage  data  collection  and  to  assist  in  the  process  of  analyzing  data 
that  may  be  either  complex  or  sparse. 

The  fourth  priority  is  an  improved  capability  to  analyze  what  will  be  complex,  high 
dimensional,  large  volume  data  effectively.  This  involves  a  combination  of  data  analysis 
techniques  and  database  management  tools  to  facilitate  handling  the  data.  A  fully  developed 
capability  will  allow  the  analyst  to  detect  and  display  cause  and  effect  relationships  and  also  to  trace 
forward  or  backward  the  strings  of  events  influencing  an  outcome.  This  capability  should  be  useful 
in  both  a  static  mode,  to  check  input  data  consistency,  and  in  a  dynamic  mode  after  run  execution. 

The  fifth  priority  is  to  provide  specialized  training  and  continuing  “mentoring”  to  analysts  in 
the  techniques  necessary  to  execute  these  more  complex  analyses.  Implementing  all  of  these 
suggestions  will  require  analysts  to  understand  a  growing  array  of  analytic  techniques,  to  be 
conversant  in  simulation  design  and  execution  issues,  and  to  be  able  to  design  and  execute  analyses 
in  distributed  environments. 

Finally,  we  see  a  need  to  continue  development  of  theory  for  new  scientific  approaches  to 
military  problems,  and  for  testing  both  the  doctrine  and  underpinning  mathematical  concepts.  More 
and  better  experimentation  will  permit  better  modeling  of  the  processes. 
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Prioritized  Model  Needs  (1  of  5) 

•  Full  spectrum  of  tools  for  the  range  of  required  analysis 

-  Warfighting  missions 

-  Non-warfighting  missions 

-  Network  centric  warfare 

-  Non-lethal:  IW,  C4ISR 

-  Non-combatant  targets  (e.g.,  financial,  educational) 

-  Multi-national 

-  Training  Evaluation  models 

•  Balanced  level  of  accuracy  to  allow  “accuracy  vs.  insight”  and  “feasibility 
vs.  optimization”  tradeoffs 

•  Ability  to  highlight  relationship  of  input  uncertainty  to  output  uncertainty 


The  needs  for  model  and  simulation  technology  parallel  the  needs  for  analytic  procedures  and 
also  address  issues  of  expanding  analytic  challenges.  The  highest  priority  need  is  for  a  full 
spectrum  of  tools  that  allow  the  analyst  to  structure  the  analysis  to  fit  the  issue  and  then  compose 
or  select  the  models  and  tools  that  are  appropriate  for  the  job. 

The  spectrum  of  tools  must  include  both  warfighting  models  and  models  and  tools  for  non¬ 
warfighting  missions.  JWARS  is  planned  to  fill  the  first  requirement,  so  it  must  be  maintained 
and  developed  into  the  future.  Other  recent  efforts  (e.g.,  MORS  OOTWAMT  Workshop,  January 
1997)  have  developed  and  identified  needs  for  tools  to  support  non- warfighting  missions.  Since 
these  missions  should  continue  to  be  a  significant  part  of  future  operations,  full  development  is 
warranted.  All  models  and  tools  should  be  useful  in  multinational  circumstances. 

To  complete  the  spectrum,  more  specialized  tools  are  required,  including  robust  C4ISR 
models,  models  for  information  warfare,  models  with  non-lethal  effects  and  models  that  address 
non-combat  targets. 
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Prioritized  Model  Needs  (2  of  5) 


•  Analyst  understanding  of  models  and  simulations 

-  It  is  important  that  the  analyst  understand  what  is  observed 

•  Analysts  need  a  means  to  determine  causality 

-  Outputs  that  the  analysts  observe  are  governed  by  the  model  logic 

•  Mapping  between  the  model  and  the  problem  space  needs  to  be  explicit 

•  Operative  objectives  and  hard/soft  constraints  must  be  clearly  identified 

•  Especially  for  reconfigurable,  interoperable  simulations,  analysts  need  a  means  to: 

-  Understand  strengths/weaknesses  of  the  simulation 

-  Establish  model  creditability 


mggjg 


The  needs  for  models  and  simulations  stem  from  the  fundamental  premise  that  analysts  need 
an  improved  ability  to  understand  what  they  observe  in  applying  modeling  and  simulation. 

In  terms  of  simulation  technology,  this  translates  to  a  need  for  model  designs  that  allow 
cause-and-effect  traceability.  Moreover,  the  analyst  should  be  able  to  easily  draw  relationships 
between  the  model  and  the  problem  space  to  be  explored.  At  present,  the  design  of  many 
analytical  models  makes  it  extremely  difficult  to  draw  relations  from  structured  variables  of  the 
model  and  the  real  world.  As  a  result,  it  is  also  difficult  to  establish  a  strong  relationship  between 
the  structure  of  the  problem  being  investigated  and  the  structure  and  response  of  the  model. 

In  a  well-structured  model  or  simulation,  the  analyst  must  be  able  to  trace  the  structure  of  the 
model  from  input  through  output  and  relate  it  to  the  problem  space.  This  capability  is  particularly 
important  for  distributed  simulations.  Technological  assistance  is  necessary  to  sort  through  and 
relate  the  various  levels  and  structures  of  the  distributed  simulations.  This  notion  is  developed 
more  in  the  next  several  slides. 
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Prioritized  Model  Needs  (3  of  5) 

•  Need  good  documentation  for  each  tool,  e.g.,  description  of  the  concept 
model,  logical  model  and  data  model 

-  Model  logic  needs  to  be  understood  in  order  to  explain  outcomes, 
recommendations  and  decisions 

-  Data  need  to  be  understood 

•  Need  simulation/analysis  tools  to  deal  with  uncertainty 

-  Less  scenario  dependent;  broader  look  at  scenario  space 

-  Broader  look  at  cultural,  political  and  human  performance  influences 

-  Methods  to  deal  with  input  data  errors  and  uncertainty  in  output 


There  is  a  fundamental  need  to  be  able  to  choose  tools,  to  structure  and  to  execute  analyses 
based  on  a  thorough  knowledge  of  the  analytic  issues  and  the  structure  of  the  tools.  Modern 
design  concepts  need  to  be  applied  to  the  array  of  analytic  tools  in  the  future.  A  well-designed 
and  documented  “Conceptual  Model  of  the  Mission  Space”  (CMMS)  will  ensure  that  the  real 
world  is  abstracted  into  a  set  of  entities  and  processes  that  aid  in  understanding  the  associated 
analytic  tools.  The  efforts  associated  with  the  JWARS  and  JSIMS  programs  provide  a  good 
foundation  in  this  area  for  combat  models. 

Furthermore,  the  implemented  tools  require  documentation  of  similar  quality  to  ensure 
thorough  understanding.  A  logic  model  of  each  tool,  making  clear  the  internal  relationships  and 
responses  of  the  model,  is  needed  to  provide  the  analyst  with  the  ability  to  explain  cause-and- 
effect  relationships.  Similarly,  data  models  need  to  be  provided  so  that  the  proper  structure  and 
pedigree  of  the  data  are  clearly  available  to  the  analyst. 

Simulation  technology  is  also  required  to  address  handling  of  uncertainty  in  analysis. 
Procedures  are  necessary  to  both  deal  with  uncertainties  within  a  single  “scenario,”  and  to  deal 
with  the  uncertainty  of  narrowly-focused  scenarios.  On  the  one  hand,  processes  to  investigate  the 
effects  of  data  uncertainty  will  provide  greater  insight  into  the  array  of  possible  results.  On  the 
other  hand,  developing  procedures  for  investigating  the  “scenario  space,”  will  provide  greater 
insight  into  the  effects  of  a  range  of  situations  and  responses. 
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Prioritized  Model  Needs  (4  of  5) 


•  Multi-mode  tools 

-  Model  uses 

•  Help  develop  understanding  (vice  requirements) 

•  Capabilities 

•  Optimization 

•  Mixes  of  above 

-  Operations 

•  Systematic  (repeatability) 

•  Interactive  (hardware  or  human  in  the  loop) 

•  Virtual 

•  Mixes  of  above 

-  Training  and  evaluation  models 


Analysts  desire  a  set  of  models  and  tools  that  are  designed  to  support  many  modes  of 
analysis.  Two  of  the  more  common  applications  are  requirements  analysis  and  capabilities 
assessments.  In  the  former,  the  analysis  seeks  to  find  what  is  necessary  in  terms  of  forces  or 
support  to  achieve  success  in  the  associated  military  operations.  Capabilities  assessments  pose  an 
order  of  battle  and  then  seek  to  determine  how  well  it  performs  in  a  given  scenario.  Both 
approaches  imply  detail  in  different  areas  of  the  supporting  model. 

A  third  analytic  approach,  optimization,  usually  requires  special  techniques  to  be  used  in 
conjunction  with  models  and  simulations.  In  these  situations,  the  objective  is  usually  to  use  the 
model  to  generate  data  on  a  variety  of  systems  or  parameters,  then  use  that  data  as  input  to  a 
mathematical  programming  model.  The  process  may  require  generating  large  amounts  of  data  on 
several  MOEs  in  a  large  run  matrix.  Besides  the  detail  necessary  for  various  systems  and 
parameters,  there  are  issues  of  managing  and  analyzing  large  data  volumes  in  batch  mode. 

The  well-developed  model  of  the  future  will  have  a  thorough  design  to  accommodate  these 
different  analytic  challenges,  as  well  as  the  supporting  structure  to  manage  and  analyze  volumes 
of  data. 

While  analytic  activities  have  generally  taken  place  through  the  systematic  use  of  closed, 
constructive  models,  analysts  need  the  ability  to  orchestrate  sound  analyses  in  both  interactive  and 
virtual  modes.  Many  issues  of  managing  reproducibility,  causal  analysis  and  statistical  procedure 
must  be  addressed  for  these  applications. 
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Prioritized  Model  Needs  (5  of  5) 

•  Potpourri 

-  Model  maintenance;  intelligent  agents  for 

•  Configuration  management 

•  Documentation 

-  Dynamic  multi-resolution 

-  Interoperable,  to  include  multi-aggregation/resolution 

-  Highly  visual,  intuitive  analysis  environments 

-  Multi-mode/dimensionality  presentations 

-  Means  to  identify/document  assumptions  imbedded  in  increasingly  complex 
models 

-  Templates  for  the  “Red  Force”  in  modeling  SSCs 


Technological  solutions  are  also  desired  to  time-consuming  and  troublesome  issues  in 
modeling  and  simulation.  One  such  problem  is  configuration  management  and  model 
documentation.  Technology  could  solve  the  problems  in  this  area  by  developing  intelligent  agents 
that  make  configuration  management  and  documentation  a  simple  process.  The  agents  would 
make  it  easy  to  search  the  structure  of  a  model,  document  what  is  found  and  make  it  easy  for  the 
analyst  to  trace  logical  flows  through  the  model. 

Technology  is  also  required  to  solve  the  problems  of  dynamic  and  multi-resolution.  Multi¬ 
resolution  allows  a  model  to  operate  at  varying  levels  of  detail.  This  capability  lets  the  model 
easily  adapt  to  different  applications,  ranging  from  course  of  action  analysis  to  weapons 
comparisons.  It  can  also  be  used  as  a  time  management  tool,  using  detail  only  where  that  detail  is 
required.  Finally,  the  ability  to  change  resolution  in  the  course  of  a  run  would  provide  maximum 
flexibility. 

In  support  of  the  more  detailed,  complex  analysis  that  will  be  possible  with  advanced  models, 
techniques  are  necessary  to  improve  the  analysis  of  outputs.  Interactive,  visual  environments  are 
high  on  the  list  of  capabilities  desired.  Also  necessary  are  techniques  to  support  analysis 
involving  highly  dimensional  data,  including  the  ability  to  trace  cause  and  effect,  and  to  parse 
marginal  effectiveness  out  among  many  contributing  factors. 

To  facilitate  faster  model  setup  and  better  analyst  understanding  of  the  new  problem  domain, 
templates  should  be  developed  similar  to  those  used  to  analyze  and  model  Soviet  forces  during  the 
Cold  War. 
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Prioritized  Data  Needs  (1  of  2) 

•  Organizations  need  to  be  identified  and  charged  with  the  responsibility  to 
create,  maintain,  certify  and  document  databases 

-  Provide  data  in  a  timely  fashion 

•  Need  automatic  scenario/da tabase  generation  tools  to  reduce  the 
resources  (time,  manpower,  dollars)  required  to  create  databases 

•  Goal  of  database  generation  in  person-days 

•  Need  a  mechanism  to  better  incorporate  lessons  learned  from  warfare, 
exercises,  AWEs,  ACTDs,  JWIDS,  tests  and  experiments  into  databases 

•  Need  a  database  help  desk 

-  “Yellow  pages”  for  data 

-  Identification  of  who  has  what  data 

-  Identification  of  how  to  get  the  data 


In  order  to  gain  maximum  value  from  the  needs  just  described,  many  improvements  are 
necessary  in  the  area  of  data  support.  The  first  change  is  to  institute  a  comprehensive 
management  process  for  data  management.  This  includes  identifying  responsible  organizations 
for  data  production  and  management.  Creation,  maintenance,  certification  and  documentation  are 
the  necessary  responsibilities  to  ensure  availability  of  timely  and  accurate  data. 

For  the  analyst,  productivity  tools  include  scenario  generation  tools.  The  objective  of  these 
enhancements  is  to  reduce  the  long  number  of  weeks  required  to  assemble  the  typical  scenario 
from  scratch.  The  tools  should  automate  activities  such  as  laying  out  order  of  battle,  setting  up 
force  flows,  arraying  forces  and  courses  of  action  and  the  like. 

Another  set  of  desired  capabilities  relates  to  the  extraction  and  assimilation  of  data  from  “real 
life”  —  including  warfare,  exercises  and  experiments  like  the  Army  Warfighting  Experiments 
(AWE)  an  Advanced  Concept  Technology  Demonstrations  (ACTD).  Extraction  and  analysis  of 
useful  data  needs  enabling  technologies. 

In  order  to  provide  efficient  access  to  these  improved  capabilities,  a  “help  desk”  is  desired. 
Characterized  as  a  “yellow  pages”  capability,  it  would  identify  who  has  data  of  what  type  and  how 
to  access  it. 
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Prioritized  Data  Needs  (2  of  2) 

•  Need  tools  (data  mining)  to  find  and  extract  the  data  that  are  needed 

•  Need  the  ability  to  define  “triggers”  that  will  monitor  the  outcomes  of  a 
model  and  collect  specified  data  when  the  triggers  are  activated 
(intelligent  agents) 

•  Applies  to  any  model 

•  Need  ability  to  define  the  output  desired  (measures,  data)  and  to  collect 
the  data  needed  for  analysis  (composable  data  collection) 

•  Need  to  make  better  use  of  tools  to  help  understand  what  the  data  have  to 
tell  (both  input  and  output  data)? 

-  Visualization 

-  Identify  and  analyze  chaotic  output. 

-  Standards  for  metadata  (source,  validation, . . .) 


Another  area  of  data  development  is  in  data  mining  tools.  Considering  the  complexity  of  data 
generated  in  the  course  of  an  exercise,  or  in  complex  analytic  runs,  it  is  understandably  hard  to 
extract  data  that  is  not  explicitly  generated.  Advanced  mining  tools  would  make  it  possible  to 
derive  relationships  across  databases,  or  derived  inferentially  from  data. 

The  next  development  desired  is  intelligent  agents  that  assist  in  generating  specialized  data. 
These  agents  would  be  triggered  by  situations  encountered  in  the  course  of  a  model  run.  Once 
triggered,  they  would  act  to  gather  detailed  data  relating  to  cause  and  effect,  providing  greater 
clarity  and  insight  for  the  analyst. 

Data  processing  procedures  to  allow  the  specification  of  desired  measures,  then  to  generate 
appropriate  data,  is  a  further  extension  of  the  data  capabilities  desired.  More  than  just  being  able 
to  generate  large  volumes  of  data  or  specified  reports,  this  capability  amounts  to  the  ability  to 
specify  a  data  type,  and  a  desired  information  level  and  to  have  the  necessary  data  gathered  for 
final  analysis  in  a  concise  process. 

Finally,  a  stronger  ability  to  extract  information  from  data  is  warranted.  The  kinds  of 
capabilities  to  aid  in  this  process  include  visualization  tools  and  processes  for  dealing  with  chaotic 
behavior. 
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Panel:  Analysis  Technology  Assessment: 
Development  Methodology 


Cateqory 

Subcateqory 

Assessment 

Comments 

Systems  Architecture/Enqineerinq 

A 

DoD  must  fund  componentware 

System  Composability 

A 

DoD  must  fund  componentware 

Scalability 

R 

Area  for  research  to  understand 
feasible  requirement.  Not  infinite 
due  to  N2  and  complexity  upper 
bounds  always  exist 

Simulation 

Frameworks/Composabilitv 

A 

DoD  must  fund  componentware 

Specification  Lanquaqe 

FJ7A 

Standards  for  Design, 
Interoperability,  and  Reuse, 

A 

Have  the  data  rules  but  not  the 
behavior  rules.  Caution;  standards 
can  impose  too  much  conformity 
and  eliminate  creative  thinkinq. 

*L«geod: 


Symbol  Meaning 

Red  (R)  Needs  exceed  aggressive  projection 

Amber  (A)  Needs  range  from 


Green  (G) 
None  (N/A) 


-  Just  below  aggressive  to 

-  Just  above  conservative 
Needs  fall  below  conservative  projection 

Not  applicable 


ynyg^j 


Systems  architecture,  composability  and  frameworks  are  interrelated  and  collectively  rated 
yellow.  Although  it  is  already  true  that  it  is  possible  to  develop  a  composable  software  system, 
there  is  no  commercial  market  for  the  componentware  needed  by  DoD  M&S.  It  will  take  DoD 
dollars  to  define,  implement  and  maintain  this  type  of  solution.  DoD  investments  are  needed  now 
to  define  and  prototype  an  efficient  component-based  system  for  M&S. 

Scalability  will  always  be  limited  technically.  It  is  arguable  that  any  software  solution  will 
have  a  scalability  limit  that  cannot  be  addressed  by  computing  power  increases  alone.  While 
technical  approaches  exist  to  mitigate  the  impact  of  scaling  up  large  software  applications,  it 
would  be  helpful  for  DoD  to  investigate  whether  it  is  possible  to  define  a  method  to  define  the 
scalability  of  an  application  at  requirements  definition  time  rather  than  discovering  scalability 
limitations  during  development. 

Standards  for  design,  interoperability  and  reuse  are  required  and  are  emerging.  It  is  observed 
that  DoD  can  go  too  far  in  imposing  standards,  the  result  of  which  is  the  stifling  of  creativity.  On 
the  other  hand,  standards  such  as  HLA  and  SEDRIS  are  important  in  the  application  domains  to 
which  they  apply.  While  it  is  argued  that  DMSO  is  properly  funded  to  develop  the  required 
standards  for  DoD  M&S,  there  is  a  concern  that  the  application  of  this  funding  may  require  added 
planning.  The  results  of  this  conference  may  be  one  input  to  a  planning  review  to  ensure  the 
available  R&D  dollars  are  focused  in  the  correct  areas. 

Sensitivity:  The  priority  for  investment  of  M&S  R&D  funds  for  development  methodology  is 
lower  than  for  modeling  methodology.  The  sensitivity  of  development  methodology  is  to  the 
overall  level  of  funding  of  M&S  in  general  and  is  directly  related  to  the  overall  funding  level  of 
M&S.  It  is  likely  that  investment  in  development  methodology  would  cease  for  significant 
reductions  in  M&S  funding.  To  a  large  degree,  funding  in  this  area  extends  existing  capability  or 
adds  capability  in  parallel  with  commercial  technology. 
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Panel:  Analysis  Technology  Assessment: 
Modeling  Methodology 


Category 

Sub-Category  |  Assessment]  Comments  ] 

Modeling 

Issues 

Cognitive 

A 

Partial,  limited  capability  only  within  a  restricted  domain  even  under  aggressive 
assumptions.  Red  if  you  require  a  complete  cognitive  model. 

H 

Red  since  the  basic  cognitive  models  are  incomplete,  modeling  the  effect  of  stress,  etc.  on  the 
cognitive  model  not  possible 

Performance  modulators 
-  Physical 

A 

Research  and  data  already  underway.  Requires  resources  for  data  collection  and 
additional  experimentation 

Physical  effects 

D 

High  resolution  models  only,  for  aggregate  see  next  subcategory.  Some  specific  physical 
effects  areas  Red 

Systems/Organizational 

effects 

A 

Dealing  with  how  entities  organize  themselves  into  larger  groups:  information  flow, 
networks,  aggregation,  etc. 

Requirements 

Process 

Analysts 

A 

Determining  MOEs,  level  of  resolution,  etc.  for  the  specific  analysis,  is  and  always  will  be 
an  an. 

Understanding 

Q 

Understanding  main  drivers  and  conceptual  space.  Efforts  such  as  CMMS  underway 

Capture 

G 

Functional  design  and  knowledge  engineering.  Efforts  underway 

Translation 

A 

Functional  design  to  systems  spec 

Other 

Model  Complexity 

A 

Current  trend  toward  models  of  increasing  complexity  makes  it  harder  for  analyst  to 
comprehend.  This  will  be  exacerbated  in  distributed  federations. 

Multidisciplinary 
skills/  teams 

A 

Resources  required  to  have  people  who  can  do  the  work;  pay  their  salaries,  pat  to  participate 
in  collaboration,  etc.  New  funding  model  may  be  required 

W&A 

A 

Harder  to  do  as  models  get  bigger  and  more  complex,  particularly  when  distributed. 

Requires  even  more  resources 

'Legend:  Symbol  Meaning 

Red  (R)  Needs  exceed  aggressive  projection 

Amber  (A)  Needs  range  from 

-  Just  below  aggressive  to 

-  Just  above  conservative 

Green  (G)  Needs  fall  below  conservative  projection 

None  (N/A)  Not  applicable 
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Panel: Analysis  Tech  Assessment: 
Computation/Communication  Capability 


Category 

Sub-Category  |  Aaaeaament  j  Com  menu  j 

J 

I 

1 

Computing 

Platform 

H 

B 

Commercial  advances  will  provide  adequate  DoD 
capability 

Computing 

Power 

y 

B 

Same  as  above:  Moore's  Law 

Networks 

B 

y 

Speed 

B 

y 

Same  as  above:  Moore's  Law 

Semp 

B 

y 

Unique  DoD  requirements  may  exceed  commercial  capabilities 

Distributed 

Computing 

y 

y 

Same  as  Confuting  platform;  Moore's  Law 

Sccurity/MLS 

D 

D 

Security  will  remain  a  risk  area 

*L«gcnd:  Symbol  Meaning 

Red  (R)  Needs  exceed  aggressive  projection 

Amber  (A)  Needs  range  from 

-  Just  below  aggressive  to 

-  Just  above  conservative 

Green  (G)  Needs  fall  below  conservative  projection 

None  (N/A)  Not  applicable 
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Panel:  Analysis  Technology  Assessment: 
Data/Information  Understanding 


Category 

Sub>Category  j  Assessment  |  Comments  j 

} 

i 

I 

Tools  for 
dealine  with 

Acquisition 

D 

y 

data 

Storage 

D 

y 

Transformation 

D 

y 

Presentation, 

Understanding, 

HQ 

y 

y 

Analysis 

y 

y 

Collaboration 

y 

y 

Other 

Ownership/Contro!  of 
Analysis  &  Data 

y 

u 

*Legend: 


Symbol 
Red  (R) 
Amber  (A) 


Green  (G) 
None  (N/A) 


Meaning 

Needs  exceed  aggressive  projection 
Needs  range  from 

-  Just  below  aggressive  to 

-  Just  above  conservative 

Needs  fall  below  conservative  projection 
Not  applicable 
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Sensitivity:  Lack  of  Commercial  Progress 


SENSITIVITY:  Commercial  Technology  docs  not  advance 


Category 

Sub-Category  |  Assessment  |  Comments  j 

Conservative 

Aggressive 

Computing 

Platform 

B 

B 

PC/woricstatioo  mix  will  be  adequate 

Computing 

Power 

D 

B 

DoD  M&S  advancements  anticipate  realization  of  Moore's  Law 

Networks 

Speed 

□ 

a 

Same  as  above 

Setup 

□ 

a 

Same  as  in  Baseline 

Distributed 

Computing 

E 

B 

Same  as  Computing  power 

Security/MLS 

□ 

n 

Commercial  advances/failures  don't  drive  DoD  capabilities 

•Legend: 


Symbol 
Red  (R) 
Amber  (A) 


Green  (G) 
None  (N/A) 


Meaning 

Needs  exceed  aggressive  projection 
Needs  range  from 

-  Just  below  aggressive  to 

-  Just  above  conservative 

Needs  fall  below  conservative  projection 
Not  applicable 
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Sensitivity  to  Assumptions 


General 

If  the  problem  domain  for  M&S  increases,  then  at  any  fixed  resource  level 
(even  if  increased),  either  each  area  gets  a  smaller  piece  of  the  pie  (more 
pieces),  or  some  areas  get  reduced  funding 

Peer 

competitor 

Technology  assessment  wont  change  but  analysis  vision  will.  This  will  tend  to 
reallocate  resources,  making  some  areas  worse  and  some  better  (depending 
upon  the  nature  of  the  threat) 

JV2010 

Since  the  analyses  should  be  able  to  represent  a  broad  variety  of  alternative 
doctrines,  success  or  failure  of  JV2010  should  not  impact  this  area 

Defense/M&S 
Budget  change 

If  the  budget  is  increased,  it  will  be  in  response  to  some  perceived  threat  or 
outside  stimulus.  The  particular  areas  most  affected  will  be  increased,  others 
the  same.  If  budget  is  cut,  the  JROC/force  structure  support  will  tend  to  be 
protected;  we  predict  this 

Commercial 
systems  failure 

May  effect  run  times,  limiting  repeated  experimentation.  Otherwise  limited 
effect 

Systems  trends 

It  is  currently  difficult  to  differentiate  between  small  system  changes.  This  will 
not  change  if  DoD  changes  procurement  emphasis 
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Recommendations  (1  of  2) 


•  The  analyst  is  most  important  and  needs  preparation  for  the  future 
environment 

-  DoD  must  increase  education  in  analysis-related  programs  for  uniformed  and 
civilian  analysts;  increase  promotion/recognition  opportunities 

-  Make  the  qualifications  of  the  analysts  a  major  factor  in  awarding  contracts  for 
outsourced  analysis. 

-  Provide  a  capstone  course  in  military  analysis  techniques 

-  Train  study  directors  of  complex  analysis  efforts  involving  distributed  tools  and 
interdisciplinary  teams 

•  Build  an  analytic,  experimental  underpinning  for  AWEs,  ATDs,  ACTDs, 
exercises  to  extract  valid  information  and  derive  usable  data 


Several  proposals  emerge  to  parallel  the  needs  just  identified.  The  first  addresses  education. 
It  is  evident  that  the  advances  in  modeling  and  analytic  technique  just  described  will  place 
unprecedented  demands  on  the  skill  of  military  analysts.  We  advocate  a  formal  educational 
course  that  trains  analysts  in  the  techniques  and  processes  involved  in  complex  analysis. 

Since  the  capabilities  will  continue  to  emerge  and  be  refined,  a  continuing  education  process 
is  also  warranted  to  keep  analysts  qualified  in  the  latest  techniques. 

In  order  to  continue  the  advancement  in  analysis,  we  also  advocate  a  program  of  continued 
research.  The  research  should  address  both  military  phenomenology  and  scientific  advancement. 
Military  phenomenology  studies  are  especially  important  in  the  context  of  future  warfighting 
approaches.  Many  of  the  scientific  disciplines  of  interest  have  already  been  suggested  in  the 
previous  discussions. 

A  particularly  important  suggestion  is  to  underpin  AWEs  and  ACTDs  with  an  analytic 
process  that  allows  valid  conclusions  and  usable  data  to  be  extracted  from  the  process.  This 
includes  both  a  solid  experimental  structure  for  the  experiments  as  well  as  analytic  procedures  to 
extract  valid  insights  from  the  volumes  of  data  generated. 

To  support  some  of  the  capabilities  already  described,  intelligent  agent  technology  should  be 
pursued. 
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Recommendations  (2  of  2) 


Support  a  program  of  research/experimentation  for 

-  Tomorrow’s  military  mission  phenomenology 

-  Application  of  emergent  science  into  analysis 

Basic  research  in  the  modeling  of  combat  needs  to  be  performed  if  the 
models  of  the  future  can  respond  to  current  and  projected  requirements 

-  Cognitive  processes  (C2) 

-  “Soft”  factors 

-  Aggregation  and  disaggregation 

-  Explicit  treatment  of  uncertainty 

-  Others... 

Data  need  as  much  attention  and  money  as  model  development  (e.g., 
JWARS,  JSIMS) 

-  New  data  needs  -  C4ISR,  distributional  information,  etc. 

-  Current  data  needs  -  releasable  information  from  Services 


Appendix  B.l  -  23 


Summary/Observations 


*  Demands  for  decision  making  information  will  continue  to  grow 

-  Complexity  and  diversity  of  the  issues  will  also  increase 

-  Information  will  increasingly  be  drawn  “all  source/any  source” 

-  The  success  of  “analysis”  in  the  future  will  depend  on  the  ability  to  prepare  for  and 
operate  in  that  environment 


In  the  year  2007,  we  believe  the  decision  making  environment  will  remain  difficult  due  to  the 
complexity  of  operational  missions  and  the  nature  of  defense  decisions  to  be  made.  Issues  will 
tend  to  be  difficult,  involving  systems  of  systems  or  components  of  systems  operating  in  uncertain 
environments  and  circumstances.  We  will  find  that  data  are  available  from  many  sources  in  the 
future,  and  we  must  be  prepared  to  exploit  it.  By  generating  the  kind  of  simulation  and  analysis 
capability  described  in  this  report,  technologists  will  help  analysts  take  a  large  step  forward  in  the 
ability  to  conduct  analysis.  However,  to  be  a  successful  application  in  the  decision  making 
environment  of  the  future,  there  will  be  a  need  for  the  ability  to  gather  and  use  disparate  data  and 
apply  the  models  we  have  described  through  this  workshop. 
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Report  of  the 

Acquisition  Working  Group 
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Participants 

Gary  Jones,  Chair,  Workshop  I 

Richard  Helmuth,  Co-Chair,  Workshop  I 

Annie  Patenaude,  Chair,  Workshop  II 

Dell  Lunceford,  Co-Chair,  Workshop  II 

Darrell  Bench 

Iris  Kameny 

Jim  Brady 

Col  Crash  Konwin 

Fritz  Brinck 

Lana  McGlynn 

Phil  Dickinson 

Peter  Merkle 

Robin  Frost 

Stephen  Olson 

Dave  Gross 

Patrick  Riley 

Marvin  Hammond 

Maj  Laurie  Rouillard 

Don  Haskell 

Greg  Searle 

Don  Hodge 

Stuart  Starr 

John  Hughes 

Dave  Thomen 

Hal  Jones 

Stephen  Upton 

We  agreed  with  the  December  definition  of  acquisition  for  the  purposes  of  the  workshop.  It 
is  more  than  that  tiny  sliver  that  lies  between  requirements  definition  and  everything  else  that 
goes  on  in  the  life-cycle  of  a  military  product.  We  have  chosen  to  define  acquisition,  for  this 
exercise,  as  not  being  separate  and  distinct  from  operation,  sustainment,  etc.  It  should  be 
thought  of  as  total  life-cycle. 
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Acquisition  Working  Group  Objective 

•  Characterize  the  acquisition  process  of  2007 

•  Identify  major  changes  that  must  occur  to  execute  the  acquisition  process 

•  Identify  shortfalls  in  M&S 

•  Prepare  actionable  recommendations  to  address  shortfalls  in  investments 
and  incentives,  policy  and  organization 


Identify  and  define  needs  and  issues  of  the  acquisition 
community  of  2007  toward  meeting  JV2010  and  other  DoD 
goals. 


The  SIMTECH II  group  re-evaluated  and  revised  the  vision,  characterization  of  M&S  in 
SBA  and  the  changes  that  need  to  occur  to  realize  the  vision. 

The  changes  were  categorized  into  cultural,  management,  policy  and  technical. 
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Acquisition  Vision  and  Goals 


An  acquisition  process  in  which  DoD  and  industry  are  enabled  by 
robust,  collaborative  use  of  simulation  technology  that  is 
integrated  across  life-cycle  phases,  functions,  and  programs  to: 

•  Substantially  reduce  the  time,  resources  and  risk  associated  with  the 
entire  life-cycle  process; 

•  Increase  the  quality,  military  worth  and  supportability  of  fielded  systems 
while  reducing  total  ownership  costs  throughout  the  life-cycle. 


Our  vision  and  goals  therefore  are  defined  here.  The  distinction  that  acquisition  is  total  life- 
cycle  support  and  not  the  traditional  early  phase  support  and  throw  it  over  the  fence.  DoD  and 
Industry  are  enabled  in  a  collaborative  environment  enabled  by  simulation  technology.  This 
environment  is  robust  in  its  breadth  and  depth  of  domain. 

The  vision  encompasses  the  collaboration  between  acquisition  phases  and  functions  within 
a  program  and  the  collaboration  between  programs. 

Functions  refer  to  acquisition  processes  that  occur  throughout  the  acquisition  process  such 
as  test  and  evaluation,  training,  combat  development  and  engineering  and  manufacturing 
development. 

Such  an  environment  must  address  time  (schedule),  resources  (people  and  funding)  and  risk 
across  the  entire  life-cycle. 

We  cannot  sacrifice  the  quality  (performance  and  support)  of  the  products  we  are 
developing  for  the  sake  of  costs.  But  we  must  control  the  total  cost  of  ownership. 

When  the  vision  is  implemented  it  will  be  a  giant  upon  which  the  program  managers  can 
stand  to  execute  their  programs. 
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Characterizations  of  the  End  State 


Increased  contractor  total  system  responsibility  with  more  efficient 
government  insight 

-  Government-industry  trusted  partnerships 

-  Shared  data  and  models 

Highly  integrated  electronic  work  environment  —  across  all  life-cycle 
functions 

-  Concurrent  development  with  continuous  evaluation  of  alternatives 

•  Ready  accommodation  of  technology  insertion/tumover 

•  Flexibility  to  accommodate  a  major  force  restructuring 

•  Integrated  training/analysis  development 

Reduced  life-cycle  costs  and  development  time  to  be  consistent  with 
commercial  practices 

DoD  commitment  to  making  the  most  informed  acquisition 
selection/decisions  based  upon: 

-  Life-cycle  cost 

-  Authoritative  data  and  model  sources 

-  Collaborative  model  and  simulation  use  _ 


-  Proper  IPT  use  —  resourced,  funded  and  empowered 


Above  are  some  of  the  characteristics  that  we  envision  for  the  SB  A  end  state. 

As  time  goes  on,  there  will  be  more  and  more  of  the  development  responsibility  shifting  to 
industry.  Partnerships  that  permit  the  sharing  of  data  and  models  must  be  developed  that  have 
trust  as  a  cornerstone  of  the  relationship. 

The  environment  will  be  an  electronic,  one  highly  integrated  across  all  life-cycle  functions. 
This  integrated  digital  environment  will  permit  the  early  and  continuous  visibility  and  evaluation 
of  system  development  alternatives.  Such  alternatives  can  include  technology  insertion  and 
turnover,  major  restructuring  of  forces  integrated  with  training. 

Typical  military  production  times  can  be  as  much  as  10-15  years.  Commercial  practices 
allow  for  much  shorter  times.  The  DoD  must  become  more  consistent  with  some  of  these 
commercial  practices.  The  difficulty  in  doing  this  is  that  the  DoD  does  not  build  large  quantities 
for  profit  and  does  not  invoke  the  incentives  of  industry. 

There  must  be  a  commitment  by  DoD  that  life-cycle  cost  is  the  basis  for  making  acquisition 
selection  decisions.  The  development  and  distribution  of  models  that  are  authoritative  needs  to 
occur.  We  must  begin  migration  toward  making  the  most  informed  acquisition  decisions  we 
can.  We  believe  that  M&S  enables  this.  IPT  must  be  taken  seriously  —  properly  resourced, 
funded  and  empowered. 

IPTs  need  to  be  trained,  intelligent  IPTs  must  be  formed  that  really  know  how  to  function. 
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Major  Changes  That  Must  Occur  To  Achieve  That 
Vision  (1  of  4) 

•  Cultural 

-  Greater  reliance  on  M&S 

-  Horizontal  integration  and  break  down  stovepipes 

-  Govemment/industry  trusted  partnerships 

-  Pervasive  sharing  of  models  and  data 

-  Up  front  investment  in  modem  processes 

-  Enabling  of  international  involvement 

-  Ability  to  conduct  comprehensive  life-cycle  trades 

-  Flexibility  to  accommodate  a  major  force  restructuring 

-  Education  for  the  vision 


There  are  four  aspects  to  the  major  changes  that  we  see  must  occur:  Culture,  management, 
policy  and  technology/environment.The  second  workshop  agreed  with  the  December  workshop 
on  the  cultural  changes. 

We  must  begin  to  move  to  greater  reliance  on  M&S  to  reduce  costs.  There  must  be 
integration  across  services  and  domains.  We  must  find  a  way  to  foster  trusted  government 
industry  partnership  to  promote  pervasive  sharing  of  models  and  data.  This  will  not  come  by 
dictate  but  will  necessitate  an  incentive  for  sharing. 

The  concepts  we  are  promoting  are  complex  and  difficult  to  comprehend.  A  program  to 
educate  future  executives,  program  mangers,  etc.  must  be  initiated  that  educates  for  the  vision. 

Incentive  for  sharing,  must  work  on  incentives;  an  example  is  the  Internet. 
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Major  Changes  That  Must  Occur  To  Achieve  That 
Vision  (2  of  4) 


•  Management 

-  Alignment  of  development  time  to  be  more  consistent  with  commercial  life-cycle 
times 

-  Ready  accommodation  of  technology  insertion/tumover 

-  Business  Process  Reengineering  (BPR)  of  data  production 

-  Functional  IPTs  are  a  way  of  life 

•  Leverage  M&S  across  functions/domains-avoid  redundancy 

-  Authoritative  sources  for  models  and  data 

•  Program  manager  provide  for  system  models 

•  Other  stakeholders  for  environment,  etc 

-  DoD  commitment  to  life-cycle  costs  basis  for  acquisition  selection/decisions 


The  incorporation  of  the  paradigm  of  SB  A  is  being  pursued  in  commercial  practice  at  a 
faster  pace  than  DoD.  These  techniques  are  providing  for  faster  cycle  times  and  reductions  in 
production  costs.  The  DoD  must  be  responsive  to  these  changes  and  accommodate  industry. 

This  paradigm  will  allow  for  major  system  modifications  and  consideration  of  significant 
alternatives  at  various  stages  of  product  development.  Management  must  be  responsive  to  the 
opportunities  provided. 
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Major  Changes  That  Must  Occur  To  Achieve  That 
Vision  (3  of  4) 


•  Policy 

-  Up  front  M&S  investment  as  the  norm  to  reduce  life-cycle  costs 

-  M&S  strategy  integral  to  the  total  acquisition  plan 

-  Models  and  simulation  critical  to  formal  acquisition  decisions 

•  Policy  guidance  on  what  the  DAB  can  expect  and  guidance  to  the  PM  on  what  should  be 
provided 

-  Incentives  for  all  stake  holders  to  participate 

-  DoD  policies  and  guidance  for  M&S  use 

-  Policy  and  guidance  on  sharing  M&S 

•  Between  government  and  industry 

•  Across  programs 


Ejgg 


The  roles  and  responsibilities  of  government  and  industry  must  be  defined  and  redefined  as 
the  barriers  and  stovepipes  in  the  traditional  life-cycle  functions  begin  to  blur  into  continuous 
integrated  functions. 

A  policy  must  be  established  that  provides  up  front  investment  in  the  capabilities  to  take  full 
advantage  of  SB  A.  A  negative  incentive  is  established  if  the  funding  for  these  efforts  are  taken 
from  the  Program  Managers  (PM).  There  must  be  a  mechanism  for  providing  incentive  for  the 
PMs  who  use  these  methods  properly  for  saving  life-cycle  costs.  This  includes  the  aspect  of 
project  funding  distribution  which  will  change  in  an  integrated  environment. 

We  must  identify  and  change  the  policies  and  guidance  that  are  inhibitors  to  making  SBA 
successful. 

How  do  we  institutionalize  M&S  to  be  a  critical  part  of  formal  acquisition  decisions? 
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Major  Changes  That  Must  Occur  To  Achieve  That 
Vision  (4  of  4) 


•  Technology/environment 

-  DoD-wide  knowledge-based  infrastructure  to  enable  SBA 

•  Program-specific  functional  integration 

•  Appropriate  utilization  of  COTS  and  GOTS 

•  Interoperability  and  reuse  standards 

-  Ability  to  comprehend  and  conduct  trades  across  highly  diverse  mission/functional 
areas 

-  Ease  of  use 

•  Cross  platform/plug  and  play/“throw  away” 

•  Rapid  initialization  and  intuitive  data  interpretation 

-  Validated  data  sources,  models  and  tools 

-  Investment  in  a  comprehensive  modeling  capability 


Data  in  the  form  of  knowledge  must  be  sharable  across  DoD,  in  and  across  programs.  At  the 
heart  of  the  SBA  capability  is  a  source  of  knowledge.  There  are  several  technologies  that  will 
enable  this:  Object  Oriented  (OO)  Data  and  Browsers.  These  will  permit  a  common  form  of  the 
data  (OO),  a  common  means  of  distribution  (the  Internet/Intranet)  and  a  common  means  of  access 
(browsers). 

The  focus  needs  to  be  on  reuse  and  interoperability  standards  to  allow  this  to  happen. 

DoD  is  not  the  leader  in  many  of  the  information  technologies  and  consequently  we  can  not 
dictate  the  form  of  developments  in  this  area.  We  must  take  advantage  of  what  Industry  is  doing, 
leverage  their  efforts  and  apply  the  solutions  to  our  problem  areas. 

The  DoD  must  prepare  itself  for  very  complex  systems-of- systems  analysis.  These  analyses 
will  require  the  advanced  technologies  offered  by  the  SBA  paradigm. 

The  implementation  of  these  technologies  are  going  to  require  large  amounts  of  information 
being  transmitted  to  a  large  number  of  distributed  sights  requiring  high  performance  high  band 
width  communications,  but  probably  only  a  fraction  of  the  time. 
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What  Shortfalls  Exist  in  Functional  Requirements 


•  Investments  and  incentives 

-  Inadequate  up  front  investment 

-  Lack  of  personal  and  programmatic  reward  for  implementing  the  vision 

-  Insufficient  sharing  of  M&S  and  data 

-  Not  enough  effective  government  and  industry  partnerships 

•  Policy  initiatives 

-  Lack  of  guidance  for  M&S  use  in  formal  acquisition  decisions 

-  Lack  of  early  and  continuous  program  focus  on  M&S  in  acquisition  life-cycle 

-  Current  policy  limitations  on  effective  use  of  commercial  domestic  and 
international  products  and  services 

-  No  common  implementation  for  sharing  M&S  and  data 

•  Organization  and  focus 

-  No  mechanism  to  ensure  compatible  processes  and  approaches 

-  Lack  of  adequate  stewardship  for  M&S  tools,  standards  and  data 

-  Lack  of  stewardship  for  M&S  education  and  training 

-  Absence  of  intellectual  foundation  for  SB  A  implementation 

•  Technical  challenges 

-  Data  availability,  quality,  credibility  and  vetting  process 

-  Interoperability  of  tools 

-  Sensitivity/security  of  M&S  and  data 

-  Physics-based,  cost,  performance  and  scheduling  models 

-  Interoperable,  variable -resolution  family  of  models 

-  Limitations  in  hardware  and  software  capabilities 


The  shortfalls  as  we  related  them  to  Acquisition: 

•  We  don’t  have  an  effective  way  to  model  what  we  need. 

•  Distributed  operations  are  increasing.  We  will  focus  on  integrated  IPT  models  to 
evaluate  the  trades  between  design  and  everything  else  (manufacturability,  testability.) 

•  We  need  models  that  are  easy  to  use 

•  Models  that  demonstrate  military  worth 

Inadequate  investment  means  inadequate  corporate  investment,  i.e.  across  programs  and 
services.  Now  a  program  will  do  what’s  right  for  it,  but  there  is  not  the  investment  to  take  the 
best  products  or  lessons  learned  across  other  programs.  There  has  to  be  investment  in  the 
technology,  not  by  individual  programs,  but  rather  by  concerted  programs.  Compatible  means 
more  than  just  common  processes. 

The  other  challenge  on  which  we  agreed  with  Workshop  I  is  the  lack  of  an  organizational 
focus  to  resolve  the  stewardship  and  direct  compatible  processes  and  approaches.  This 
management  could  also  include  an  executive  training  team  for  SB  A. 
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Actionable  Recommendations  to  Remedy  High 
Priority  Shortfalls  (1  of  2) 


•  Investment  and  incentives 

-  Establish  and  support  sufficient  M&S-infrastructure  investments  in  the  POM 

-  Provide  incentives  to  all  stakeholders  accompanied  by  adequate  up  front  investment  for  M&S 
use  early  and  throughout  the  life-cycle  to: 

-  Minimize  total  cost  of  ownership. 

•  Shorten  acquisition  cycle  time 

•  Improve  support  to  the  warfighters  and  decision  makers 

-  Incentivize  active  partnering  between 

•  Acquisition  programs 

•  Government  and  industry 

•  Policy  initiatives 

-  Establish  policy/guidance  to  address  M&S  use  in  formal  acquisition  decisions 

-  Direct  Requirements  developers  and  SAEs  be  held  accountable  to: 

•  Maximize  SB  A  benefits 

•  Reduce  life-cycle  costs 

-  Define  a  policy  to  utilize  emerging  products  and  services  both  commercial  domestic  and 
international  to  maximize  SB  A  potential 

-  Establish  DoD  policy  and  a  common  implementation  for  sharing  M&S  and  data 

•  Organization  and  focus 

-  Identify  and  empower  an  organization  to  enable: 

•  Dedicated  enduring  pilot  and  flagship  programs 

•  Stewardship  for  managing  SB  A 
—  Focus  by  priority  what  needs  to  be  done 


Org  and  Focus:  It  is  beginning  to  become  very  widely  accepted  that  there  is  enormous 
potential  to  reduce  total  life-cycle  costs  and  maintain  system  performance  through  the  proper 
construction,  distribution,  access  and  use  of  system-of-systems  knowledge. 

SBA  partnering  needs  to  be  done  on  many  levels: 

•  Top  level  through  M&S  Executive  agents  working  with  Industry  conglomerates. 

•  PEOs  working  with  domain  specific  industry. 

•  PMs  working  with  prime  contractors. 

Dr  Gansler’s  letter  to  the  services  on  SBA  underscores  his  commitment  to  this  effort  but 
doesn’t  put  a  hook  in  place  to  insure  that  this  is  done. 

We  agreed  that  there  needs  to  be  a  single  point  to  manage  SBA,  not  a  council  or  a  policy 
organization  but  someone  empowered  to  make  the  needed  decisions  with  the  horsepower  to 
back  them  up. 
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Actionable  Recommendations  to  Remedy  High 
Priority  Shortfalls  (2  of  2) 


•  Technologies 

Modeling  methodologies  are  the  most  serious  shortfalls  that  require  high  priority  for 
funding 


Issue 

Do  What 

Representation 

DMSO  identify  as  a  high  priority  in 
the  next  version  of  the  DoD  M&S 

Master  Plan 

Simulation  Techniques 

Community  working  on  CGF 
should  reprioritize  and  put  the 
effort  into  new  techniques 

Levels  of  Abstraction 

DoD  needs  to  consider  link 

between  CAD/CAM  and  operational  effectiveness 
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Sensitivity  Analysis  (1  of  3) 


Mission  trends 

-  A  peer  competitor  emerges  prior  to  2007  [minor] 
more  likely  to  affect  what  we  buy,  not  how 

Strategy/operational  trends  [minor] 

-  Joint  Vision  2010 

•  Perceived  as  successful  (i.e.,  new  joint  concepts  of  operation  are  adopted) 

•  Perceived  as  unsuccessful  (i.e.,  return  to  more  deliberate/mass  oriented  operations) 
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Sensitivity  Analysis  (2  of  3) 


•  Institutional  trends 

-  “Top-line”  defense  budget:  +/- 20%  from  the  baseline 

-  [Moderate/Major]  If  there  is  a  top-line  20%  reduction,  this  would  have  a  favorable 
impact  because  it  would  force  people  to  use  SBA 

-  M&S  expenditures:  +/- 20%  from  the  baseline  [Moderate/Major]  A  reduction 
would  adversely  impact  SBA 

♦  Systems  trends 

-  Limited  acquisition  resources  prompts  DoD  to  plan  to  retain  systems  for  50% 
longer  than  existing  systems  [Moderate  +]  For  systems  retained  longer,  there  will 
be  opportunities  for  ROI  using  SBA  in  the  O&S  tail 

-  DoD  shifts  to  an  acquisition  strategy  that  emphasizes  more,  cheaper  (possibly 
lower  tech)  systems  [Moderate  +] 

-  SBA  would  allow  for  analysis  of  more  complex  system 

-  of  systems 
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Sensitivity  Analysis  (3  of  3) 


Commercial  trends 

-  Reliance  on  commercial  products  proves  to  be  a  failure  (e.g.,  commercial  products 
fail  to  provide  adequate  environmental  robustness  or  respond  adequately  to  DoD’s 
unique  needs),  prompting  DoD  to  rely  less  on  commercial  products,  services 
[Moderate  +].  SB  A  will  be  helped  -  it  will  aid  in  being  more  selective  in  the 
response 

-  Acquisition  reform/RBA  fails  to  deliver  (e.g.,  acquisition  reform  proves 
disappointing)  [Major-] 

-  This  will  have  the  effect  of  cutting  off  support  for  SBA 
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Executive  Summary  (1  of  2) 
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The  past  decade  has  seen  Modeling  and  Simulation  (M&S)  in  support  of  education  and  training  move  from 
concept  to  reality.  Ten  years  ago,  almost  to  the  day,  the  concept  of  distributed  simulation  in  support  of  training  was 
being  developed  at  the  Warrior  Preparation  Center.  Since  that  time,  this  revolutionary  concept  has  been  proven  and 
has  grown  to  be  commonplace.  During  the  same  time  period,  the  foundations  for  distributed  education  were  formed 
with  numerous  courses  being  given  at  every  education  level. 

The  benefits  that  M&S  and  its  associated  technologies  bring  to  education  and  training  are  no  longer  in  doubt. 

The  initial  technology  base  of  inexpensive  high  bandwidth  communications,  software  engineering,  high  performance 
computing  and  multi-media  presentation  tools  coupled  with  the  explosive  growth  of  the  Internet  is  in  place  and 
improving  daily. 

The  next  ten  years  will  no  doubt  see  equally  significant  changes.  It  is  the  purpose  of  this  panel  on  education  and 
training  to  attempt  to  define  the  operational  requirements  for  military  education  and  training  for  the  next  decade.  The 
follow-on  workshop  will  attempt  to  define  the  M&S  capabilities  required  to  meet  these  requirements 

The  primary  focus  of  this  workshop  has  been  constrained  to  focus  largely  on  training  and  education  in  support  of 
developing  skills  for  the  military  and  civilian  warfighting  force.  Little  discussion  took  place  over  the  equally 
important  problems  associated  with  educating  and  training  personnel  responsible  for  either  the  acquisition  of  these 
forces  or  the  policy  side  of  DoD  operations  due  to  the  time  constraints.  It  is  clear  that  these  other  areas  deserve  equal 
attention  and  that  the  ultimate  solutions  are  interconnected  and  synergistic. 

The  world  has  changed  from  a  bi-polar  threat  environment,  focused  on  high  intensity  large  force  combat,  to  a 
non-polar  environment  in  which  the  military  is  employed  across  a  range  of  operations  from  humanitarian  operations  to 
counter-terrorism  to  combat  against  potential  threats  in  virtually  any  region  of  the  world.  As  a  result,  the  doctrine 
employed  by  the  military  is  in  the  process  of  the  most  significant  changes  in  the  last  50  years.  All  of  the  services  have 
programs  (the  Army  After  Next,  the  Alternative  Air  Forces,  etc)  that  are  in  the  process  of  defining  the  organizations, 
weapons  systems  and  doctrine  that  will  keep  them  as  dominant  in  the  next  20  years  as  they  were  in  Desert  Storm 
despite  significant  drawdown,  increased  PERSTEMPO  and  OPTEMPO  and  increased  breadth  of  operations.  Given  this 
change,  education  and  training  will  play  a  more  significant  role  than  ever  before. 

Multiple  technologies  are  likely  to  emerge  that  will  be  critical  for  training  applications.  An  exhaustive  list  is  not 
provided  here,  but  some  of  the  key  technical  areas  area.  Improvements  in  communications  and  computer  processing 
should  enable  long-distance  leaming/training  applications  such  as  ADL.  Web-based  technologies  such  as  Java  should 
help  to  provide  ease  of  application  sharing  across  platforms  that  will  lead  to  improvements  in  DL  and  ADS.  Advances 
in  AI  will  hopefully  lead  to  improvements  in  human  behavioral  representation  —  a  key  to  improving  interactive 
training  with  simulations. 

It  is  critical  that  we  begin  to  exploit  these  new  technologies  as  soon  as  possible.  The  rapid  evolution  of 
technology  will  tend  to  make  detailed,  long-to-develop  plans  obsolete  before  they  can  be  implemented.  Instead,  we 
should  begin  to  focus  on  how  we  can  utilize  emerging  technologies  and  gain  experience  in  applying  them  that  can  be 
built  upon. 
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Executive  Summary  (2  of  2) 


The  current  training  and  education  process  cannot  maintain  pace  with  the  rapid  change  in  technology. 

Technology  changes  demands  a  rapid,  flexible  process  that  can  educate  the  warfighter  quickly  and  satisfy  a  variety  of 
technical  capabilities.  A  complete  revision  to  the  training  process  must  take  place  that  can  accommodate  these 
changing  needs. 

The  influx  of  new  technologies  for  training  will  also  require  new  methods  of  training  in  applying  them.  We  must 
adopt  methods  that  will  help  us  change  the  WAY  we  leam  in  addition  to  WHAT  we  learn.  New  learning  methods  that 
stress  the  ability  to  assimilate  information  will  likely  be  required,  instead  of  traditional  methods  that  focused  on 
memorization  or  repetition. 

Training  and  education  within  the  services  and  across  DoD  is  believed  to  suffer  from  a  lack  of  coordination. 

While  efforts  exist  that  are  taking  advantage  of  new  technologies  to  improve  training  and  education,  these  efforts  may 
not  be  widely  known  or  synchronized.  It  is  recommended  that  a  periodic  “needs  assessment  be  performed  which 
would  (1)  identify  shortfalls  in  the  training  and  education  domains;  (2)  prioritize  these  needs  and  fund  efforts  to  correct 
them  via  an  implementation  plan;  and,  (3)  develop  a  feedback  process  that  will  periodically  revise  this  implementation 
plan. 

Currently,  the  responsibility  for  education  falls  primarily  on  the  service  institutions.  However,  with  the  trend  in 
downsizing  across  the  DoD  it  is  expected  that  institutions  will  have  fewer  resources  upon  which  to  draw  to  implement 
education  and  training.  Individuals  must  take  more  of  the  responsibility  for  training  and  educating  themselves.  DoD 
must  adopt  a  policy  that  will  provide  incentives  for  individuals  to  improve  themselves  through  education  and  training. 
Likewise,  institutions  must  share  in  this  process  so  that  available  resources  are  not  squandered. 

Changes  in  the  personnel  management  system  must  be  made.  Current  practices  involve  training  individuals  early 
in  their  careers  with  the  requisite  knowledge  that  they  will  need.  However,  the  rapid  evolution  of  technology  often 
makes  knowledge  obsolete  within  only  a  few  years.  Therefore,  the  personnel  systems  must  accommodate  the  need  for 
continuous  training  throughout  the  career  cycle.  The  system  must  make  allowances  for  continuous  training  and 
education  throughout  one’s  career  —  not  just  “front! oaded”  at  the  beginning. 

To  facilitate  this,  broadbased  training  must  be  integrated  with  specific,  tailored  training  throughout  a  soldier  s 
career.  An  example  of  broadbased  training  might  be  knowledge  of  basic  computer  operations  that  could  be  used  to 
gain  knowledge  when  necessary.  This  would  allow  the  user  to  access  more  specific  information  as  necessary.  Links  to 
non-military  (i.e.,  universities,  community  colleges,  etc.)  will  be  necessary  to  expand  the  knowledge  base  for  such 
information. 

In  summary,  this  workshop  established  operational  requirements  that  help  to  define  the  technologies  to  be 
addressed  in  the  next  workshop.  In  addition  to  the  requirements  for  changes  to  be  made  in  training  and  education 
systems,  numerous  policy  changes  must  be  made  to  implement  processes  that  make  effective  the  technology  changes. 
The  bottom  line  is  that  the  last  decade  laid  the  foundations  for  the  necessary  technologies  to  “revolutionize”  education 
and  training;  however,  the  foundations  for  the  necessary  policy  changes  still  need  to  be  put  in  place. 
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Why  Are  We  Here? 


Workgroup  Objective 

“After  consideration  of  the  lessons 
learned  from  the  retrospective 
assessment,  the  participants  will 
identify  and  prioritize  the  needs  of 
the  users  of  military  M&S” 


“If  we  continue  to  develop  our 
technology  without  wisdom  or 
prudence,  our  servant  may  prove  to  be 
our  executioner 
-  General  Omar  Bradley 


The  past  decade  has  seen  Modeling  and  Simulation  (M&S)  in  support  of  education  and 
training  move  from  concept  to  reality.  Ten  years  ago,  almost  to  the  day,  the  concept  of 
distributed  simulation  in  support  of  training  was  being  developed  at  the  Warrior  Preparation 
Center.  Since  that  time,  this  revolutionary  concept  has  been  proven  and  has  grown  to  be 
commonplace.  During  the  same  time  period,  the  foundations  for  distributed  education  were 
formed  with  numerous  courses  being  given  at  every  education  level. 

The  benefits  that  M&S  and  its  associated  technologies  bring  to  education  and  training  are 
no  longer  in  doubt.  The  initial  technology  base  of  inexpensive  high  bandwidth 
communications,  software  engineering,  high  performance  computing  and  multi-media 
presentation  tools  coupled  with  the  explosive  growth  of  the  Internet  is  in  place  and  improving 
daily. 

The  next  ten  years  will  no  doubt  see  equally  significant  changes.  It  is  the  purpose  of  this 
panel  on  education  and  training  to  attempt  to  define  the  operational  requirements  for  military 
education  and  training  for  the  next  decade.  The  follow-on  workshop  will  attempt  to  define  the 
M&S  capabilities  required  to  meet  these  requirements. 

The  primary  focus  of  this  workshop  has  been  constrained  to  focus  largely  on  training  and 
education  in  support  of  developing  skills  for  the  military  and  civilian  warfighting  force.  Little 
discussion  took  place  over  the  equally  important  problems  associated  with  educating  and 
training  personnel  responsible  for  either  the  acquisition  of  these  forces  or  the  policy  side  of 
DoD  operations  due  to  time  constraints.  It  is  clear  that  these  other  areas  deserve  equal 
attention  and  that  the  ultimate  solutions  are  interconnected  and  synergistic. 
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Revolution  in  Military  Affairs 


Attrition  Maneuver  Full  Spectrum 

Warfare  Warfare  of  Missions 


The  Background: 

The  world  has  changed  from  a  bi-polar  threat  environment,  focused  on  high  intensity  large 
force  combat,  to  a  non-polar  environment  in  which  the  military  is  employed  across  a  range  of 
operations  from  humanitarian  operations  to  counter-terrorism  to  combat  against  potential 
threats  in  virtually  any  region  of  the  world.  As  a  result,  the  doctrine  employed  by  the  military 
is  in  the  process  of  the  most  significant  changes  in  the  last  50  years.  All  of  the  services  have 
programs  (the  Army  After  Next,  the  Alternative  Air  Forces,  etc)  that  are  in  the  process  of 
defining  the  organizations,  weapons  systems  and  doctrine  that  will  keep  them  as  dominant  in 
the  next  20  years  as  they  were  in  Desert  Storm  despite  significant  drawdown,  increased 
PERSTEMPO  and  OPTEMPO  and  increased  breadth  of  operations. 

Given  this  change,  education  and  training  will  play  a  more  significant  role  than  ever 
before. 
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Trends  Affecting  Training  (1  of  2) 


•  More  complex  military  requirements 

-  Expanded  mission  space  (US  and  foreign) 

•  More  diverse  threats 

•  Expanded  emphasis  on  non-combat  operations 

•  Expanded  operations  with  non-DoD  and  non-govemmental 

-  Coalition  and  combined  operations 

•  Increased  operations  and  decision  space  with  reduced  decision  time 

-  More  complex  equipment 

-  Increased  OPTEMPO 

-  Continued  force  downsizing 

•  Casualties  and  collateral  damage  unacceptable 


jjjjgjgj 


The  military  faces  a  significantly  expanded  mission  space:  threats  ranging  from  drug 
dealers  to  terrorists  to  major  regional  powers.  By  the  end  of  the  next  decade,  it  is  likely  that 
major  powers  such  as  China  will  arise  that  are  capable  of  posing  serious  threats  to  the  US.  In 
addition  to  combat  operations,  the  military  is  expected  to  provide  non-combat  support  around 
the  world  in  combination  with  both  non-DoD  and  non-government  organizations. 

With  the  increased  mission  space,  comes  increased  operations  space  as  a  result  of 
weapons  systems  that  can  engage  multiple  targets  at  extended  ranges,  of  weapons  systems  that 
can  travel  and  fight  at  significantly  increased  speeds,  and  C4ISR  systems  that  can  provide 
huge  volumes  of  high  resolution  information  about  both  the  military  and  cultural  environment. 
Coupled  with  this  increased  operations  space,  comes  increased  decision  space  with  reduced 
decision  time.  More  options  to  explore,  high  OPTEMPO  conflict,  broader  threat  spectrum  all 
result  in  less  time  to  make  decisions. 

Making  the  problem  even  more  complex  is  the  political  impact  of  casualties  or  collateral 
damage  on  the  employment  of  military  forces.  Numerous  examples  of  political  impact  range 
from  Somalia  to  Desert  Storm. 
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Trends  Affecting  Training  (2  of  2) 


•  Tailored  military  organizations 

-  Force  composition  and  C2  change  with  military  requirements 

-  More  complex  organizations 

•  Increased  reachback  operations 

•  Common  operational  picture 

•  HCI/Human-human  interface  —  increased  requirement  for  collaboration 

•  Merging  of  training,  planning  and  military  operations 

-  Merging  of  M&S  and  C2  systems 

-  Embedded  analyses  in  training  events 

-  Use  of  instrumented  training  events  to  provide  data  to  support  other  domains 

•  Outsourcing 
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In  support  of  the  expanded  mission,  operations  and  decision  space  and  the  significantly 
smaller  force,  the  structure  of  the  forces  deployed  has  changed  significantly.  Smaller  forces, 
increasingly  joint,  tailored  to  specific  missions  with  significant  support  remaining  in  the 
CONUS  or  theater  rear  areas  are  becoming  the  norm.  The  result  is  a  force  that  depends  on 
precision  and  mobility  to  achieve  “overwhelming  force”  with  minimum  casualties  and 
collateral  damage. 

The  result  of  these  factors  is  a  significant  increase  in  the  requirement  for  a  more  highly 
educated  and  trained  force  that  can  be  successful  with  fewer  forces  across  a  broader  range  of 
operations  in  a  highly  visible  battlefield. 
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The  current  process  has  a  number  of  elements:  In-residence/correspondence/seminar 
PME;  hands-on,  informal  apprenticeship;  and  military  exercises.  The  consequence  of  stove- 
piped  education  and  training  is  a  significant  limitation  in  our  ability  to  merge  the  pillars  of  JV 
2010  and  execute  the  vision  of  “Full  Spectrum  Dominance.”  In  other  words,  if  we  don’t 
“revolutionize”  our  education  and  training  processes  we  won’t  be  able  to  “revolutionize”  our 
military  capabilities. 
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Current  Education  and  Training  Process 


Little  Relationship  Between  Formal  Education 
and  Training  Courses  and  Military  Operations 


Formal  Training 
Courses 

In-Residence 
Education 


In  general,  while  each  element  imparts  critical  knowledge  in  and  of  itself;  they  are 
independent  and  not  linked  together  thus  unable  to  provide  the  degree  of  either  growth  or 
sustainment  that  a  well  structured-well  integrated  program  might  bring. 

In  addition,  and  perhaps  more  critical,  the  periods  of  peak  performance  only 
coincidentally  match  military  requirements  because  the  training  and  education  events  are  fixed 
events  pushed  to  the  military  member  rather  than  a  resource  able  to  be  pulled  when  required. 
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Future  Education  and  Training  Requirements  (1  of  2) 


•  Valid  training  metrics  and  methodology 

-  Define  training  requirements  and  performance  standards 

-  Training  effectiveness  measures 

-  Using  results  to  refine  and  improve  the  process  (diagnose,  fix  and  improve) 

•  More  efficient  knowledge  and  skill  transfer 

-  Limited  training  time  and  resources  available 

-  More  information  transfer  required 

-  Accurate  determination  of  current  level  of  knowledge 

-  Training  tailored  to  individual’ s  and  team’s  most  efficient  learning  method 
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Given  the  smaller  forces  (downsizing);  the  increased  OPTEMPO/PERSTEMPO;  and  the 
increasingly  complex  world  around  us,  the  education  and  training  process  must  be 
significantly  more  efficient  —  it  must  transfer  much  more  knowledge  in  much  less  time. 

Additionally,  the  education  and  training  process  must  be  a  life  long  process  because  of  the 
rapidly  changing  world  and  the  need  for  each  person  to  be  proficient  in  more  skills  (fewer 
people  —  more  complex  work). 

Lastly,  the  training/education  must  come  to  the  people  not  the  people  to  it  (increased 
OPTEMPO/PERSTEMPO).  Two  technologies  are  in  their  infancy  which  if  developed  and 
integrated  under  the  Advanced  Distributed  Learning  (ADL)  initiative,  hold  much  promise  to 
meet  these  demands:  ADS  (growing  out  of  distributed  wargaming  capabilities  like  the  Joint 
Training  Confederation  and  STOW)  and  Distance  Learning  (DL)  coupled  with  the  rapidly 
growing  commercial  entertainment  market. 
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Future  Education  and  Training  Requirements  (2  of  2) 


•  Lifelong  training 

-  Greater  breadth  of  required  skills 

-  More  frequent  skill  changes  required 

-  Must  train  with  non-DoD  agencies 

•  Continuous  training  opportunity 

-  Distance  learning 

-  Advanced  distributed  simulation 
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Future  Education  and  Training  Vision  (1  of  3) 

•  Just-in-time/just-enough  training  tailored  to  requirements  and  delivered 
when  and  where  required 

-  Education  and  training  mutually  supportive 

-  Education  &  training  research  expanded 

-  Leveraging  Non-DoD  technology  advances 

•  Non-DoD  assets 

•  Stimulate  commercial  market  place 

•  Teaching  “how  to  learn”  will  become  as  important  as  training  “how  to 
do” 

•  Use  of  simulation  for  discovery  learning 

•  Training,  analysis  and  acquisition  communities  will  provide  mutual 
support  and  exploit  common  resources 


Egg 


The  vision  of  just-in-time/just-enough  training  distributed  when  and  where  required  is 
self-explanatory. 

It’s  important  to  realize  that  this  capability  represents  a  major  paradigm  shift  in  how  we 
currently  train  and  educate  with  DoD.  This  shift  increases  the  responsibility  for  learning  from 
the  institution  to  the  individual  and  requires  a  significant  focus  on  teaching  “how  to  learn”  and 
thus  an  increased  focus  on  the  science  of  education.  As  the  individual  becomes  responsible  for 
learning,  the  institutions  become  responsible  for  creating  a  learning  environment  and  providing 
both  infrastructure  and  content. 

The  result  could  look  something  like  the  next  slide. 
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Future  Education  and  Training  Vision  (2  of  3) 


An  Education  and  Training  Cloud: 

The  individual  can  draw  upon  a  distributed  education/training  cloud  fed  by  the  institutions 
and  available  to  the  ‘lifelong  student’  when  and  where  he  requires  it.  (A  pull  system  support 
just-in-time/just-enough  training.) 

The  in-residence  programs  will  still  be  vital,  but  will  shift  focus  to  teaching  “learning 
processes”  and  group  skills  and  provide  the  “glue”  that  feeds  knowledge  across  elements. 

Producing  this  change  in  the  current  stovepiped  education  and  training  process  will 
produce  the  following  outcome: 
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Future  Education  and  Training  Vision  (3  of  3) 


A  fully  integrated,  constantly  available  education  and  training  system  that  supports  “pull” 
as  well  as  “push”  —  fully  supporting  meeting  the  goals  of  JV  2010. 
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Future  Education  and  Training  Process 


Direct  Relationship  Between  Formal  Education 
and  Training  Courses  and  Military  Operations 


A  continuous  growth  in  capabilities  with  new  skills  acquired  based  on  military  needs: 
new  jobs  (assignments)  or  military  events  requiring  new  skills. 

The  goal  is  to  transition  today’s  classroom  focused  education  process  and  unstructured 
hands-on  training  into  a  continuously  available  distributed  system  composed  of  courses  and 
on-line  knowledge  bases  that  can  be  pulled  by  the  “life  long  student”  as  required,  giving  each 
individual  the  resources  to  become  an  “expert”  when  he  or  she  needs  to  be. 
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Observations 


•  ‘Without  data  we  are  nothing”  LaBerge  ‘87 

•  Past  10  years  built  technology  building  blocks  —  we  must  develop  a 
system  that  will  exploit  current  and  future  technology 

•  Education  and  training  science  requires  extension  and  exploitation 

•  Rate  of  growth  in  technology  outpaces  ability  to  exploit  it  —  institutional 
change  required  to  take  maximum  advantage  of  emerging  technologies 

•  Full  exploitation  of  the  available  opportunities  requires  significant 
cultural  change  within  DoD.  (Most  critical  issue) 

•  Exploitation  lags  new  technology  despite  ever  increasing  requirements 

•  Total  force  training  must  be  consistent 


The  key  observations  from  the  workshop  focused  around  the  fact  that  it  is  not  technology 
that  is  holding  back  the  education  and  training  revolution.  It  is:  1)  a  lack  of  a  digital 
knowledge  base  (data);  2)  inadequate  linkage  between  educational  science  (how  we  learn  and 
how  to  train);  3)  the  lack  of  linkage  between  military  educational  institutions  and  the  training 
systems;  and,  4)  the  lack  of  clear  personal  responsibility  for  education  and  training  and  a 
supporting  system  to  make  it  possible. 

To  put  in  place  a  “revolutionized”  education  and  training  system  requires  significant 
institutional  change  accomplished  through  major  policy  changes. 
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Policy  Requirements  (1  of  2) 

•  Establish  comprehensive  educational  process  synchronized  across  DoD 

-  Perform  “needs  assessment” 

-  Develop  education  action  plan  for  DoD  education  and  training 

-  Develop  a  comprehensive  training  methodology 

-  Develop  comprehensive  feedback  process 

•  Responsibility  for  education  and  training  shared  between  institutions  and 
individuals 

-  Adopt  policy  to  “incentivize”  individuals 

-  Adopt  policy  to  “incentivize”  institutions 

•  Develop  T&E  experimentation  plan  within  the  scope  of  the  JV  2010 
experimentation  campaign  plan 


yyjgjgj 


Training  and  education  within  the  services  and  across  DoD  is  believed  to  suffer  from  a 
lack  of  coordination.  While  efforts  exist  that  are  taking  advantage  of  new  technologies  to 
improve  training  and  education,  these  efforts  may  not  be  widely  known  or  synchronized.  It  is 
recommended  that  a  periodic  “needs  assessment”  be  performed  which  would  (1)  identify 
shortfalls  in  the  training  and  education  domains;  (2)  prioritize  these  needs  and  fund  efforts  to 
correct  them  via  an  implementation  plan;  and,  (3)  develop  a  feedback  process  that  will 
periodically  revise  this  implementation  plan. 

Currently,  the  responsibility  for  education  falls  primarily  on  the  service  institutions. 
However,  with  the  trend  in  downsizing  across  the  DoD  it  is  expected  that  institutions  will  have 
fewer  resources  upon  which  to  draw  to  implement  education  and  training.  Individuals  must 
take  more  of  the  responsibility  for  training  and  educating  themselves.  DoD  must  adopt  a 
policy  that  will  provide  incentives  for  individuals  to  improve  themselves  through  education 
and  training.  Likewise,  institutions  must  share  in  this  process  so  that  available  resources  are 
not  squandered. 
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Policy  Requirements  (2  of  2) 


•  Improvement  in  analysis  training 

-  User  training  in  analysis  to  leverage  integrated  decision  support  /  C2  systems  (what 
ifing) 

-  Increased  Ops  Analysts  Training  in  Battlestaff  Support 

•  Create  ADL  and  ADS  program  office  to  integrate  education/training 
technologies 

•  Increase  EXCIMS  focus  on  integration/synthesis/convergence  across 
training/analysis/acquisition  M&S  programs 

•  Integrate  training  system  development  into  systems  acquisition 


As  decision  support  tools  are  integrated  with  command  and  control  systems,  there  will  be 
a  need  to  improve  the  training  of  the  users  of  such  tools  in  traditional  analysis  techniques.  The 
use  of  decision  support  tools  to  formulate  Courses  Of  Action  (CO As),  perform  wargaming 
excursions  and  define  strategy  must  be  based  upon  a  fundamental  knowledge  of  analysis. 
Otherwise,  such  tools  may  be  misused. 

The  advent  of  new  technologies  will  have  a  profound  effect  on  training  and  education. 
Distance  Learning  (DL)  and  Advanced  Distributed  Simulation  (ADS)  have  the  potential  to 
revolutionize  the  training  and  education  process.  Such  tools  will  be  able  to  place  a  user  in 
contact  with  geographically  dispersed  knowledge  bases  and  provide  new  sets  of  simulation 
tools  that  may  speed  learning  and  improve  retention.  Because  this  technology  is  so  pervasive, 
it  may  be  prudent  to  oversee  the  application  of  DL  and  ADS  through  the  formation  of  a 
program  office  that  can  coordinate  the  implementation  across  all  of  DoD  (implementation  of 
the  advanced  distributed  learning  initiative). 

There  are  still  redundant  applications  of  simulation  across  the  training,  analysis,  and 
acquisition  areas  of  DoD.  To  the  extent  possible,  the  EXCIMS  should  focus  on  removing  the 
redundancies  in  those  areas  and  leverage  off  similar  efforts.  It  is  recognized  that  some  efforts 
in  these  domains  are  unique  and  cannot  be  leveraged,  but  may  be  able  to  utilize  common  core 
M&S  pieces. 
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Management  Requirements  (1  of  2) 


•  Revise  training  process 

-  Changes  in  Personnel  Management  System  must  be  made  —  can’t  count  on 
providing  most  of  the  training  on  accession  —  must  train  continuously  — 
periodically  return  to  schoolhouse 

-  Need  to  integrate  broad  based  training  with  tailored  training  conducted  concurrent 
with  operations. 

•  Technology  changing  too  rapidly;  learning  curve  too  long  and  specialized  (example: 

SONAR  technician  with  highly  specialized  skills) 

•  Differentiate  training  based-upon  retention  requirements  (short-time  vs.  long  term) 

-  Links  to  academic  (non-military)  knowledge  sources  (universities)  — 
knowledge/training  base  much  expanded 


The  current  training  and  education  process  cannot  maintain  pace  with  the  rapid  change  in 
technology.  Technology  changes  demands  a  rapid,  flexible  process  that  can  educate  the 
warfighter  quickly  and  satisfy  a  variety  of  technical  capabilities.  A  complete  revision  to  the 
training  process  must  take  place  that  can  accommodate  these  changing  needs. 

Changes  in  the  personnel  management  system  must  be  made.  Current  practices  involve 
training  individuals  early  in  their  careers  with  the  requisite  knowledge  that  they  will  need. 
However,  the  rapid  evolution  of  technology  often  makes  knowledge  obsolete  within  only  a 
few  years.  Therefore,  the  personnel  systems  must  accommodate  the  need  for  continuous 
training  throughout  the  career  cycle.  The  system  must  make  allowances  for  continuous 
training  and  education  throughout  one’s  career  —  not  just  “frontloaded”  at  the  beginning. 

To  facilitate  this,  broadbased  training  must  be  integrated  with  specific,  tailored  training 
throughout  a  soldier’s  career.  An  example  of  broadbased  training  might  be  knowledge  of 
basic  computer  operations  that  could  be  used  to  gain  knowledge  when  necessary.  This  would 
allow  the  user  to  access  more  specific  information  as  necessary.  Links  to  non-military 
establishments  (universities,  community  colleges,  etc.)  will  be  necessary  to  expand  the 
knowledge  base  for  such  information. 
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Management  Requirements  (2  of  2) 


•  Improved  synergy  between  analysis,  acquisition  and  training  domains 

-  Use  of  common  resources  (simulators  for  prototyping,  determining  requirements 
and  training,  etc) 

-  Change  mindset  that  training  objectives  are  orthogonal  to  analysis/acquisition 
objectives 

•  Shared  data  collection  between  domains  without  repercussions  to  individuals 

•  Multi-discipline  distributed  labs 

•  Education/training  stovepipes  must  be  integrated 

-  Increase  synergy  between  schoolhouses  (education)  and  training  community 

•  ADL  links  with  institutional  schoolhouses 


Management  of  the  analysis,  acquisition  and  training  domains  must  be  done  in  such  a  way 
that  they  may  share  common  resources  when  necessary.  Such  examples  might  include  the 
sharing  of  simulators  (for  performing  virtual  prototyping  in  the  acquisition  domain, 
determining  requirements  in  the  analysis  domain  and  training  operators  in  the  training 
domain).  This  will  require  a  change  in  the  mindset  that  these  domains  are  orthogonal  to  one 
another.  Data  must  be  easily  accessible  and  shared  across  domains,  without  the  traditional 
concerns  of  releasability  of  data.  The  introduction  of  multi-discipline  labs  that  perform 
analysis,  training  and  acquisition  tasking  may  help  improve  synergy  across  these  domains. 

Education  and  training  stovepipes  must  be  better  integrated  in  order  to  support  the 
warfighter.  Currently  there  is  little  synergy  between  the  education  one  receives  at  the  service 
schoolhouses  and  the  training  one  receives  during  operations.  Again,  the  rapid  pace  of 
technology  demands  that  the  education  one  receives  directly  supports  the  training  one 
maintains  during  operations.  ADL  could  help  to  integrate  these  two  areas. 
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Technical  Requirements  (1  of  8) 

•  (1)  Create/adopt  “open”  architectures  which  allows  integration  of 

COTS/GOTS  products  (HLA?) 

-  Leverage  use  of  products  under  development  in  commercial  industry  (Marine 
DOOM,  Harpoon  97,  etc.) 

-  Use  of  multi-national  commercial  products 

-  (DM  Amber  -  DoD  influence  needed) 


It  is  expected  that  the  commercial  world  will  continue  to  improve  upon  simulation 
applications  for  the  entertainment  industry  over  the  next  decade.  Currently,  some  of  these 
applications  are  already  approaching  and  surpassing  the  fidelity  of  many  military  simulations. 
In  order  for  the  military  to  leverage  off  of  these  developments,  it  will  be  critical  that  they 
conform  to  the  same  simulation  architecture  that  has  been  adopted  for  interoperability  of  all 
DoD  simulations  (HLA).  If  the  commercial  world  adopts  a  different  architecture,  it  will  limit 
the  utility  of  such  simulations.  Such  an  architecture  must  be  open  to  multi-national 
commercial  products,  as  it  is  likely  that  vendors  from  various  countries  will  develop 
simulations  that  will  be  useful. 

Multiple  technologies  are  likely  to  emerge  that  will  be  critical  for  training  applications. 
An  exhaustive  list  is  not  provided  here,  but  some  of  the  key  technical  areas  are  likely  to 
include  the  following.  Improvements  in  communications  and  computer  processing  should 
enable  long-distance  leaming/training  applications  such  as  ADL.  Web-based  technologies 
such  as  Java  should  help  to  provide  ease  of  application  sharing  across  platforms  that  will  lead 
to  improvements  in  ADL  and  ADS.  Advances  in  AI  will  hopefully  lead  to  improvements  in 
human  behavioral  representation  —  a  key  to  improving  interactive  training  with  simulations. 
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Technical  Requirements  (2  of  8) 


•  (2)  Utilize  emerging  capabilities 

-  (2a)  Communications  (CC  Green) 

•  Bandwidth  (however,  bandwidth  is  commercially  driven  and  is  NOT  limitless) 

•  Accessibility 

•  Latency  (see  speed  of  light. . . ) 

-  (2b)  Computer  processing  (CC  Green) 

-  (2c)  Web-based  technologies  (Java)  (CC  Green) 

-  Interactive  Learning 

•  (2d)  Will  technology  be  there  to  support  it?  (IU  Amber) 

•  (2e)  Adapting  to  personal  learning  styles/Distributed  Learning  (Learning  theory)  (MM 
Red) 

-  (2f)  Behavioral  representation  (MM  Red) 
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Technical  Requirements  (3  of  8) 


•  (3)  Digital  libraries/repository 

-  (3a)  Data/Information  Acquisition  (1U  Red) 

-  (3b)  Storage  (IU  Amber) 

-  (3c)  Accessibility  (IU  Amber) 

•  (4)  Security  issues  must  be  addressed  with  increased  availability  (CC 
Amber) 

•  (5)  Need  to  model  societal  effects  —  political,  economic,  social  issues;  non- 
combative  extensions  (multi-national,  multi-cultural  issue)  (MM  Red) 


The  commercial  and  academic  environments  will  likely  lead  the  way  in  the  application  of 
ADL  technologies  to  promote  improvements  in  learning.  In  addition  to  satellite  classes,  future 
students  may  be  able  to  dial  up  instructional  material  on  demand  —  allowing  them  to  tailor 
their  education  needs  as  necessary.  Such  technology  could  extend  to  military  applications  in 
which  soldiers  require  specific  knowledge  to  complete  a  task  and  retrieve  it  as  required. 

In  order  to  maximize  the  benefits  of  such  technology,  standards  for  interoperability  will 
be  necessary  between  DL  applications,  as  well  as  with  ADS  applications.  Since  accessing 
information  from  a  distance  may  also  involve  running  simulations  to  better  understand 
concepts,  an  integrated  architecture  between  DL  and  ADS  will  be  required. 

With  the  advent  of  ADL  technologies,  access  to  information  will  become  increasingly 
easy.  However,  if  sensitive  information  is  also  accessible  to  enemy  interests  it  could  easily 
undermine  military  operations.  Therefore,  it  is  expected  that  new  security  methods  (multi¬ 
level  security,  improved  encryption  technologies)  will  be  necessary  with  the  development  of 
such  applications. 
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Technical  Requirements  (4  of  8) 


•  (6)  Collaboration  tools  exist  (IU  Green) 

•  (7)  Develop  processes  for  collaboration  tool  adoption  and  use  (IU  Amber) 

•  (8)  Integration  of  advanced  HCI  technology  (IU  Green) 

•  (9)  Standardization  of  HCI  across  military  systems  (IU  Red) 
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Technical  Requirements  (5  of  8) 


•  (10)  Adopt  computing  platforms  driven  by  commercial  standards  (CC 
Green) 

•  (11)  Requirements  process  (the  analysis,  understanding,  capture  and 
translation  to  system  spec)  needs  to  be  reusable  (MM  Amber) 

•  (12)  CGF  application  of  MM  techniques  to  solve  problem  of  representing 
the  virtual  world  (XX  Red) 
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Technical  Requirements  (6  of  8) 


•  (13)  Need  to  unify  operational,  systems  and  technical  architectures 
between  C4I,  M&S  and  education  and  training  communities  to  provide  a 
continuum  from  training,  planning,  mission  rehearsal  and  execution 

•  (13a)  Include  training  specialists  in  the  acquisition  BPT  (maintenance, 
technical  support  and  functional  end  user  [individual  and  collective]) 
(MM  Amber) 

•  (13b)  Reduce  training  time/resources  required  for  mastery  for  both 
trainees  and  trainers  (IU  Amber) 

-  (13c)  Common  infrastructure  and  standard  interfaces  to  plug  and  play  (DM  Amber) 

-  (13d)  Train  on  systems  we  fight  on  (When  training  on  the  real  system  is  the  best 
way  to  train)  (IU  Amber) 


A  major  technical  goal  of  the  next  ten  years  will  be  the  utilization  of  an  architecture  that 
allows  interoperability  between  C4I  systems  and  M&S.  Current  technologies  do  not  support 
such  interoperability,  without  unique  hardware  (man-in-the-loop  in  many  cases)  and  software. 
The  goal  within  the  next  decade  should  be  to  allow  the  majority  of  military  C4I  systems  to 
“plug-in”  to  the  majority  of  military  M&S  applications  and  exchange  information  without 
having  to  build  unique  interfaces.  This  will  promote  the  ability  to  train  warfighters  on  the 
same  C4I  systems  that  they  will  use  in  the  field,  at  reduced  costs  for  specialized  interfaces  to 
models. 

Another  M&S  training  requirement  over  the  next  decade  will  be  the  collection  of 
performance  data  on  training  exercises.  Rather  than  collecting  vast  amounts  of  data  without 
regard  to  which  data  is  important,  a  strategy  of  determining  the  relevant  MOEs  and  MOPs 
prior  to  a  training  event  is  recommended. 
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Technical  Requirements  (7  of  8) 


•  (14)  Need  to  collect  data  on  training  performance 

-  (14a)  Should  be  theory  driven  (don’t  collect  data  for  sake  of  collection)  (IU  Red) 

-  (14b)  Need  for  instrumentation  of  systems  (technical  and  human)  to  collect 
meaningful  quantitative  and  qualitative  measures  of  training 
efficiency/effectiveness  (IU  Red) 

-  (14c)  Operational  community  must  allow  training  effectiveness  to  be  measured 
with  individual/team  attribution)  (IU  Red) 
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Technical  Requirements  (8  of  8) 


•  14  (Continued) 

-  (14d)  Need  increase  measurability  of  existing  performance  standards 
(UJTL/Mission  Essential  Tasks)  (IU  Amber) 

-  (14e)  Develop  equivalent  performance  standards  for  the  acquisition  and  analysis 
communities  (IU  Amber) 

-  (140  Need  to  develop  training  effectiveness  measurement  tools  (IU  Amber) 
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Equally  Great  Thoughts... 


•  Need  to  promote  research  in  new  training  methodologies 

•  Need  to  respond  to  new  training  methodologies  with  new  training 
technologies 

-  New  technologies  support  an  expanded  training  space 

•  Train  skills  we  can’t  train  today 

•  Use  training  methods  not  possible  today 

•  Don’t  wait  to  begin  to  exploit,  otherwise  will  never  get  there 


The  influx  of  new  technologies  for  training  will  also  require  new  methods  of  training  in 
applying  them.  We  must  adopt  methods  that  will  help  us  change  the  WAY  we  learn  in 
addition  to  what  we  learn.  New  learning  methods  that  stress  the  ability  to  assimilate 
information  will  likely  be  required,  instead  of  traditional  methods  that  focused  on 
memorization  or  repetition. 

It  is  critical  that  we  begin  to  exploit  these  new  technologies  as  soon  as  possible.  The  rapid 
evolution  of  technology  will  tend  to  make  detailed,  long-to-develop  plans  obsolete  before  they 
can  be  implemented.  Instead,  we  should  begin  to  focus  on  how  we  can  utilize  emerging 
technologies  and  gain  experience  in  applying  them  that  can  be  built  upon. 


Appendix  B.3  -  30 


Technology  Assessment  (1  of  2) 
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We  need  to  define  it  first 

m 

Adapting  to  personal  teaming  stytes/Distributed  Learning  (Learning 
theory) 

EM 

Behavioral  representation 

No  viable  HBR  theory 

Digital  Libraries/Repository 

em 

Data/Information  Acquisition 

No  structured  program 

EM 

Storage 

E9 

No  one  is  funding  containers 

a 

Accessibility 

S3 

Creating  infrastructure  and 
mechanisms  to  retrieve  info 

■■i 

— 
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Technology  Assessment  (2  of  2) 


rrE  m/issue 

ESI 

a 

COMMENTS 

7 

Develop  processes  lor  collaboration  tool  adoption  and  use 

MH 

■■ 

u 

Cultural  barriers;  How  to  best  use  collaboration 

8 

Integration  ol  advanced  HCI  technology 

^■1 

555 

Driven  by  commercial  industry 

9 

Standardization  ol  HCI  across  military  systems 

£351 

55 

Not  standardized;  Cultural  barriers 

10 

Adopt  computing  platforms  driven  by  commercial  standards 

_ 

3LUJI 

H 

a 

12 

CGF  application  ol  MM  techniques  to  solve  problem  of 
representing  the  virtual  world 

No  DoD  program;  No  complete  theory  lor  HBR 

13“ 

Need  to  unify  operational,  systems  and  technical  architectures 
between  C4I,  MAS  and  Education  &  Training  communities  to 
provide  a  continuum  from  Training.  Planning,  Mission 

Rehearsal  and  Execution... 

i 

1 

1 

1 

Indude  training  specialists  in  the  acquisition  IPT  (maintenance, 
technical  support  and  functional  end  user  (individual  and 
collective]) 

A 

■ 

■ 

Significant  cultural  barriers 

s 

Reduce  training  time/resources  required  for  mastery  tor  both 
trainees  and  trainers 

A 

Lack  of  training  theory 

■ 

Common  infrastructure  and  standard  interfaces  to  plug  and 
play 

a 

■ 

Evolution  of  systemitechnology  increases 
requirement;  Technology  green,  Operations 
amber 

13d 

Train  on  systems  we  fight  on  (When  training  on  the  real  system 

is  the  best  way  to  train) 

A 

Not  always  best  training  available;  Resources 
often  unavailable  due  to  OPS 

14 

Need  to  collect  data  on  training  performance 

■ 

a 

Should  be  theory  driven  (don't  collect  data  tor  sake  of 
collection) 

I 

In’.  1 5? IL v JL*i OTiiLiI.* t AiiETiiiDaCaL Si  .l  — 

■ 

■ 

■ 

No  structured  program;  no  theory  for  collection 

■ 

Operational  community  must  allow  training  effectiveness  to  be 
measured  with  indhnduairteam  attrbirtjon  (on  a  not-to-interfere- 
with-operations  basis) 

■ 

■ 

No  structured  program;  no  theory  lor  collection 

El 

Moving  in  that  direction 

ii 

Develop  equivalent  performance  standards  for  the  acquisition 
and  analysis  communities 

A 

Moving  in  that  direction 

tnm 

Need  to  develop  training  effectiveness  measurement  tools 

. . 

A 

Moving  in  that  director 
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John  Gilmore 
Richard  Haruyama 
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Bob  Orlov 
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Marcy  Stahl 
Glenn  Tarbox 
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Expert  Opinions 


Predicting  the  distant  future  is  easy.  Solar  power  will  be  harnessed,  cures 
will  be  found  for  present-day  diseases,  and  the  New  York  Mets  will  win  the 
World  Series.  The  real  difficulty  is  predicting  tomorrow  —  figuring  out  the 
next  step  in  getting  from  where  we  are  to  where  we  are  going. . ..Arthur  Miller 
of  Motorola  Inc.  in  IEEE  Spectrum  June,  1994 

Always  listen  to  the  experts.  They’ll  tell  you  what  can’t  be  done  and  why  - 
then  do  it. . .  .Robert  A  Heinlien 

A  conclusion  is  the  place  where  you  got 
tired  of  thinking.... Unknown 

Gjggg 


At  best,  predicting  the  future  is  problematic.  At  worst,  it  is  a  waste  of  time. 

Consequently,  we  will  not  attempt  to  predict  the  future,  but  rather  attempt  to  make  bounding 
estimates  concerning  where  technology  might  be  10  years  from  now  given  certain  assumptions 
about  ongoing  DoD  spending  and  the  directions  that  may  be  taken  by  commercial  information 
technology  development. 
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Definitions* 


•  Conceptual  Model:  A  statement  of  the  content  and  internal 
representations  which  are  the  user's  and  developer's  combined  concept  of 
the  model.  It  includes  logic  and  algorithms  and  explicitly  recognizes 
assumptions  and  limitations. 

•  Model:  A  physical,  mathematical  or  otherwise  logical  representation  of  a 
system,  entity,  phenomenon  or  process. 

•  Simulation:  A  method  for  implementing  a  model  over  time. 

•  Modeling  and  Simulation  (M&S):  The  use  of  models,  including  emulators, 
prototypes,  simulators  and  stimulators,  either  statically  or  over  time,  to 
develop  data  as  a  basis  for  making  managerial  or  technical  decisions.  The 
terms  "modeling**  and  "simulation"  are  often  used  interchangeably. 


These  foundational  definitions  are  provided  to  aid  in  this  presentation  on  modeling  and 
simulation  technology 
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Conservative  and  Aggressive  Prediction 
-  Definitions  -- 


•  Conservative  Predictions:  Assumes  that  the  area  maintains  its  current 
trend* 

•  Aggressive  Predictions:  Assumes  that  the  trend  for  the  area  increases 
substantially* 


*  SaieK  U  V*D  m  fuww uct*u 


InlgsE* 

■HP 


Our  conservative  predictions  assume  that  current  priorities  and  funding  levels  for  M&S 
technologies  remain  constant  and  that  the  current  rate  of  improvement  in  M&S  technology 
development  continues. 

Our  aggressive  predictions  assume  that  the  area  of  M&S  technology  receives  substantially 
higher  priority  and  funding.  Also,  we  assume  that  the  rates  of  advance  in  M&S  technology 
increase  because  of  increased  funding  or  because  of  unexpected  breakthroughs. 
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SIMTECH  2007  M&S  Technology  Taxonomy 


Theories,  processes, 
algorithms,  and  information 
for  conceptual  modeling 


Techniques,  tools  and  software 
for  design,  construction  and 
implementation 


The  formulation  of  this  taxonomy  of  M&S  technology  was  undertaken  to  support 
SIMTECH  2007.  It  is  not  necessarily  a  decomposition  that  would  be  preferred  for  other  types 
of  examinations  of  information  application  domains  and  their  related  technology.  These  top- 
level  bins  are  are  further  divided  into  subcategories  to  support  the  detailed  deliberations  of 
SIMTECH  2007. 

Modeling  methodology:  Includes  the  theories,  processes,  algorithms  and  information  that 
support  the  conceptualization  of  a  model. 

Development  methodology:  Includes  the  techniques,  tools  and  software  used  in  the 
design,  construction  and  implementation  of  models  or  model  tool  sets. 

Data  and  information  technology:  Includes  the  processes  and  tools  for  the  capture, 
storage,  transformation  and  understanding  of  data  and  information. 

Computation  and  communications  capability:  Includes  the  systems  on  which  the  M&S 
application  will  be  hosted  and  how  the  M&S  application  connects  to  other  M&S  applications. 
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Current  State  of  Simulation  Technology 

•  Modeling  methodology 

-  Most  recent  focus  on  high  resolution,  entity  based  technologies 

-  Simulations  still  to  hard  to  use 

•  Design  methodology 

-  Simulations  =  software; 

-  Standards  are  limited,  most  standards  in  use  are  commercial 

•  Computational  and  communications  capability 
-  A  limiting  factor,  but  not  by  much 

•  Data  and  information  technology 

-  Analysis  is  an  art,  not  a  science 

-  Collaboration  tools  are  maturing 

|rn€R3£ 

Modeling  methodology:  Most  recent  focus  is  on  high-resolution,  entity  based,  human-in- 
the-loop  technologies;  there  is  little  real  investment  in  other  areas.  There  are  few  to  no 
advances  in  Computer  Generated  Force  (CGF)  technology.  Simulations  are  still  hard  to  use 
and  it  is  too  hard  to  understand  the  results.  Users  becoming  more  comfortable,  but  still  very 
reserved. 

Design  methodology:  Simulation  =  software;  nothing  more  to  say.  Large  scale 
simulation  systems  exceed  the  communities’  ability  to  specify  and  build.  Standards  (especially 
for  interoperability  and  reuse)  limited  to  HLA  and  SEDRIS;  most  standards  in  use  are  ‘owned’ 
by  the  commercial  sector. 

Computational  and  communications  capability:  A  limiting  factor,  but  not  by  much. 

Data  and  information  technology:  No  efforts  to  instrument  and  collect  raw  data. 
Scenario  generation  is  a  limiting  factor.  Analysis  is  an  art,  not  a  science.  Collaboration  tools 
are  maturing  and  they  are  beginning  to  help  requirements  management. 
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Modeling  Methodology 


•  The  theories,  processes,  algorithms  and  information  that  support  the 
conceptualization  of  a  model 

-  Representations;  e.g., 

•  Cognitive 

•  Physical  Effects 

•  Decision  support  (C2) 

-  Simulation  techniques;  e.g., 

•  Computer  Generated  Force  (CGF)  techniques 

•  Requirements 

-  Requirements  analysis,  capture  and  understanding 

-  Translation  of  system  requirements  to  system  specification 


The  area  of  Modeling  Methodology  includes: 

Representations: 

Cognitive  (e.g.,  perception,  planning,  reasoning,  understanding) 

Classical  warfare;  emerging  warfare  areas  (e.g.,  information  warfare,  asymmetrical  warfare, 
peace  keeping  operations,  small  unit  operations) 

Performance  modulators  (e.g.,  fear,  fatigue,  skill  level) 

Physical  effects  (e.g.,  environmental  representation,  movement,  attrition,  resource 
consumption) 

Decision  support  (C2),  doctrine,  organizational  modeling 

Simulation  Techniques: 

Computer  generated  forces  techniques,  the  “how”  of  the  “what.”  The  modeling 
methodologists  need  to  understand  the  implementation  concept. 

Requirements: 

Requirements  analysis,  capture  and  understanding 
Translation  of  system  requirements  to  system  specification 
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Modeling  Methodology 
~  Conservative  Predictions 


•  Representations: 

-  Cognitive/modulators 

•  Approaches  remain  ad  hoc 

•  Limited  advances  in  emerging  warfare  areas 

-  Physical: 

•  Increasing  fidelity  in  these  areas 

•  Limited  ability  to  model  C2 

•  Simulation  techniques 

-  Few  new  AI  techniques  developed;  no  real  advances  in  CGF  technology 

•  Requirements 

-  Analysis,  capture  and  understanding 

•  Continues  to  be  underfunded 

•  Still  difficult;  essentially  a  manual  process 


Representations:  Business  as  usual:  under-funded/-emphasized/-prioritized 

Cognitive/modulators:  Approaches  remain  ad  hoc,  fragmented  and  uncoordinated. 
There  are  limited  advances  in  emerging  warfare  areas.  Research  on  human  perception  is 
limited  (adversarial  modeling). 

Physical:  Effects  and  environmental  representation:  improves,  increasing  fidelity  in  these 
areas,  yet  pieces  are  not  fully  integrated.  There  is  limited  ability  to  model  C2  structures, 
doctrine  and  tactics. 

Simulation  techniques:  Few  new  AI  techniques  are  developed;  there  are  no  real 
advances  in  CGF  technology. 

Requirements:  Analysis,  capture  and  understanding.  Continues  to  be  underfunded. 
Simultaneously,  this  becomes  a  harder  issue  because  of  changes  in  the  world  (e.g.,  very 
limited  ability  to  model  non-conventional  warfare).  The  translation  of  requirements  to  system 
specifications  is  still  difficult;  essentially  a  manual  process. 
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Modeling  Methodology 
Aggressive  Predictions  (1  of  2) 


•  Representations 

-  Representation  of  cognitive/modulators:  Program,  as  recommended  in  SIMTCCH 
97,  is  undertaken,  giving  rise  to  coherent,  community-accepted  models 

-  Perception  modeling/adversarial  modeling  matures  (but  not  perfect) 

-  Physical  effects:  Matures  to  point  conventional  and  emerging  warfare  effectively 
covered 


Representation  of  cognitive  processes  and  modulators  progresses.  The  program,  as 
recommended  in  SIMTECH  97,  is  undertaken,  giving  rise  to  coherent,  community -accepted 
models. 

Perception  modeling  and  adversarial  modeling  matures,  but  not  perfect 

Modeling  moves  in  one  of  two  directions;  large  scale,  general  purpose  simulations 
become  institutionalized,  or  boutique’  models  becomes  the  norm.  If  the  latter,  component 
interoperability  will  be  required. 

The  modeling  of  physical  effects  matures  to  the  point  that  conventional  and  emerging 
warfare  are  effectively  covered.  Tools  to  generate  geo- typical  (but  geo-specific 
representative)  will  mature  and  generation  of  geo-specific  representations  is  fast  but  still 
expensive. 
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Modeling  Methodology 
-  Aggressive  Predictions  (2  of  2) 


•  Simulation  techniques 

-  Variety  of  new  innovative  technologies  matured;  full-up,  general  purpose  CGF 
probably  not  yet  fully  realized 

•  Requirements 

-  Analysis  and  understanding  of  underlying  “physics” 

-  Translation  of  requirements  to  system  specification 


Simulation  techniques:  A  variety  of  new  innovative  technologies  matured,  although 
full-up,  general  purpose  CGFs  are  probably  not  yet  fully  realized.  There  is  reduced  emphasis 
on  knowledge  acquisition  and  knowledge  engineering. 

There  is  increased  computational  efficiency. 

There  are  significant  advances  in  the  ability  to  model  human-to-human  interactions. 

Requirements:  The  analysis  and  understanding  of  underlying  “physics”  becomes  a 
priority  issue.  Investment  in  this  area  is  increased  substantially,  however,  this  area  remains  an 
art  form  vice  a  science. 

The  translation  of  requirements  into  system  specification  experiences  breakthroughs  that 
support  automated  linking  of  requirements  to  system  specifications. 
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Development  Methodology 


•  The  techniques,  tools  and  software  used  in  the  architecting,  design, 
construction  and  implementation  of  the  models  or  the  model  tool  set 

-  System  composability  (e.g.,  Object-Oriented,  component  technology) 

-  Simulation  frameworks  and  componentware  (e.g.,  SPEEDES,  SIMSCR1PT, 
Flames,  DEVS) 

-  Standards  for  design,  interoperability  and  reuse 


Development  Methodology  includes  the  techniques,  tools,  and  software  used  in  the 
architecting,  design,  construction  and  implementation  of  the  models  or  the  model  tool  set. 
This  area  includes  concepts  such  as: 

System  composability  which  is  achieved  through  the  use  of  object-oriented  and 
component  technology. 

Simulation  frameworks  and  componentware  such  as  SPEEDES,  SIMSCRIPT,  Flames, 
DEVS. 

Standards  for  design,  interoperability  and  reuse. 
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Development  Methodology 
-  Conservative  Predictions  -- 

•  Composable  technology 

-  Continues  to  develop 

•  Simulation  frameworks 

-  Continue  to  develop  slowly;  system  level  programmers  still  control  programs 

•  Standards 

-  Largely  commercially  driven  (e.g.,  VRML) 

-  Interoperability  and  reuse  of  military  simulations  remains  difficult 

lfll£i£d 

Composable  technology  will  continue  to  develop.  For  example,  there  will  be  more  use  of 
object-oriented  techniques  for  model  construction  and  there  will  be  maturing  of  component 
technology. 

Simulation  frameworks  will  continue  to  develop  slowly.  System  level  programmers  (C++ 
programmers)  still  control  our  programs.  There  are  no  new  simulation  or  simulation  friendly 
languages  on  the  horizon. 

Standards  are  largely  commercially  driven  (e.g.,  VRML).  Interoperability  and  reuse  of 
military  simulations  is  aided  by  the  continued  maturing  of  HLA  and  SEDRIS.  However,  there 
are  no  new  standards  in  areas  such  as  CGF  behaviors.  Software  reuse  on  a  significant  scale 
continues  to  be  limited  and  difficult  to  achieve. 
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Development  Methodology 
-  Aggressive  Predictions  - 


•  Composable  technology 

-  Development  leads  to  iconic  model  construction 

-  ‘Boutique’  simulation  becomes  viable 

•  Simulation  frameworks 

-  Use  becomes  commonplace;  will  aid  interoperability  and  reuse 

•  Standards 

-  Still  commercially  driven, 

-  Interoperability  and  reuse  see  some  new  work  beyond  HLA  and  SEDRIS 


Composable  technology  development  leads  to  iconic  model  construction.  This  enables 
more  reuse  across  functional  areas.  True  rapid  prototyping  and  ‘boutique’  simulation  become 
viable. 

The  use  of  simulation  frameworks  becomes  commonplace.  This  will  aid  interoperability 
and  reuse,  but  only  within  systems  using  the  same  frameworks  except  as  enabled  by 
component  technologies. 

Standards  are  still  commercially  driven,  new  standards  may  evolve  out  of  communication/ 
entertainment  communities. 

Interoperability  and  reuse  are  advanced  by  some  new  work  beyond  HLA  and  SEDRIS, 
however,  because  of  the  time  lag  in  standards  there  is  little  impact  in  the  speed-up  of 
development  and  deployment  over  the  next  10  years. 
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Computational  and  Communications  Capability 


•  What  the  M&S  application  will  be  hosted  on,  how  it  connects  to  other 
M&S  applications  and  how  M&S  application  developers  and  users 
connect  to  one  another 

•  Computing  platform 

-  Computing  power 

-  Network  Connectivity  (e.g.,  bandwidth  requirements  for  complex  simulations) 

-  Security  (e.g.,  security  protocols) 


The  area  of  computational  and  communications  capability  includes  the  system  that  will 
host  the  M&S  application,  how  it  connects  to  other  M&S  applications  and  how  M&S 
application  developers  and  users  connect  to  one  another. 

This  area  addresses  the  computing  platform  computational  power.  Also  included  are  the 
network  and  the  associated  bandwidth  requirements  for  complex  simulations;  the  tools  for 
setting  up  and  maintaining  the  network;  the  simulation  or  simulator  building  techniques  for 
distributed  computing;  and  system  security,  for  example,  security  protocols,  especially  for 
multi-level  security. 
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Computation  and  Communications 
-  Conservative  Predictions  (1  of  2)  • 


•  Computing  platform: 

-  Still  mainly  PC/ workstation  environment 

-  However,  wireless  computing  begins  to  be  more  common 

•  Computing  power 

-  Increase  on  average  twice  as  fast  as  Moore’s  Law 

•  Network 

-  Backbone  is  10  Giga-bit 

-  Setting  up  a  network  continues  to  be  a  complex,  difficult  task 


Computing  platforms  are  still  mainly  PC  and  UNIX  workstation  environment,  however, 
wireless  computing  begins  to  be  more  common. 

Computing  power  sees  average  increases  at  a  rate  that  is  twice  as  fast  as  Moore  s  Law. 

Network  backbones  will  be  10  Giga-bit  (OC192).  The  next  generation  Internet  protocol 
emerges.  T1  connectivity  to  the  home  is  the  norm.  However,  setting  up  a  network  continues 
to  be  a  complex  and  difficult  task. 

There  are  reductions  in  the  manpower  required  to  develop  networking  solutions.  There 
are  reduction  in  costs  associated  with  hardware  for  routers  and  leased  phoned  lines.  New 
techniques  emerge  to  reduce  M&S  setup  time  and  to  reduce  network  resource  requirements. 
Solid  solutions  will  exist  for  connecting  multiple  computers  and  multiple  computer  users  to 
support  collaborative  work  (e.g.,  ATM,  multi-casting,  Internet  model). 
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Computation  and  Communications 
~  Conservative  Predictions  (2  of  2)  - 


•  Distributed  computing 

-  More  common  place  —  networkable  computers  increases  computing  power 

-  Distributed  simulation  bandwidth  needs  still  require  high  speed,  expensive 
networks 

•  Security: 

-  Concerns  begin  to  be  addressed;  still  fall  short 

-  (e.g.,  multi-level  security  is  not  achieved) 
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Distributed  computing  becomes  more  common  place,  networkable  computers  increase 
computing  power.  Distributed  simulation  bandwidth  is  still  an  issue,  high  speed  expensive 
networks  are  still  required. 

Security  concerns  begin  to  be  addressed  but  still  fall  short,  for  example,  multi-level 
security  is  not  achieved. 
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Computation  and  Communications 
-  Aggressive  Predictions  (1  of  2)  - 


•  Computing  platform 

-  Workstations  disappear,  PC-based  systems  the  norm 

•  Computing  power 

-  Grows  at  a  rate  at  least  four  times  Moore’s  Law 

•  Networks 

-  Backbone  is  500  Giga-bit 

-  Difficulty  in  establishing  a  network  considerably  improved 

-  Some  improvement  in  simulation  bandwidth  efficiency 

TKtnc 
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The  typical  computing  platform  is  a  light  weight  PC-based  system  and  workstations  have 
disappeared.  Nomadic,  wireless,  personal  data  assistants  are  in  regular  use. 

Computing  power  grows  at  a  rate  at  least  four  times  Moore’s  Law. 

The  network  backbone  is  500  Giga-bit.  There  is  good  bandwidth  to  overseas  locations, 
making  CONUS-to-global  distributed  simulation  much  more  accessible.  There  is  considerable 
improvement  in  alleviating  the  difficulty  in  establishing  a  network.  Automatic  establishment 
of  a  network  (networks  on-demand)  is  close  but  not  yet  available. 

There  are  some  improvement  in  simulation  bandwidth  efficiency,  but  due  to  cost  and 
availability  of  networks  this  area  has  low  priority. 
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Computation  and  Communications 
—  Aggressive  Predictions  (2  of  2)  — 


•  Distributed  computing 

-  Networks,  computers,  simulation  systems  are  much  more  distributed  and 
computing  friendly 

•  Security 

-  Adaptive  security  software  developed  to  address  many  or  most  security  concerns 

-  However,  perfect  solutions  still  do  not  exist 
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Distributed  computing  is  greatly  aided  by  networks,  computers  and  simulation  systems 
that  are  more  distributed  and  computing  friendly.  The  decision  to  operate  local-only  or 
distributed  is  not  based  on  difficulty  of  setting  up  and  running  distributed  exercises. 

Solutions  will  be  developed  for  the  global  addressing  problem  of  data  distribution  within 
simulation  transport  infrastructure  aiding  HLA  RTI,  SPEEDES  and  other  interoperability 
mechanisms. 

Communication  requirements  between  distributed  components  will  be  drastically  reduced. 
Techniques  will  be  developed  for  performing  a  priori  resource  requirements  analysis  to 
support  pre-exercise  load  balancing  of  simulation  objects  across  computation  resources. 

Security  solutions  for  multi-level  simulation  exercises  emerge.  Adaptive  security 
software  is  developed  to  address  many  or  most  security  concerns.  Users  are  not  faced  with 
obstacles  such  as  rebooting,  switching  hard  disks  or  continually  downloading  patches  to 
security  system  software.  However,  this  is  a  very  difficult  area  and  perfect  solutions  still  do 
not  exist. 
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Data  and  Information  Technology 


•  The  processes  and  tools  needed: 

-  To  acquire  and  transform  data/information 

-  For  data/information  storage  and  retrieval 

-  To  understand  the  model  and  its  results 

•  Includes  processes  and  tools  for: 

-  Data  acquisition,  capture  and  storage 

-  Transforming  the  data  into  useful  applications  (e.g.,  scenario  generation) 

-  Presenting,  understanding  and  collaborating  on  the  M&S  application  and  its  results 


The  area  of  data  and  information  technology  includes  the  processes  and  tools  needed  to 
acquire  and  transform  data  and  information  and  for  data  and  information  storage  and  retrieval. 
This  includes  data  for  model  creation  and  reuse  libraries  for  simulation,  model,  scenario  and 
AAR  and  Analysis  data. 

This  area  also  includes  the  handling  of  data  and  information  needed  to  understand  models, 
and  the  results  of  their  use. 

Also  included  are  the  processes  and  tools:  for  data  acquisition,  capture  and  storage;  for 
transformation  of  the  data  into  useful  applications,  for  example,  scenario  generation;  and  for 
presentation,  understanding  and  collaboration  on  the  M&S  application  and  its  results.  This  last 
area  includes:  the  Human  Computer  Interface  (HCI)  (how  the  application  appears  and  how 
you  interact  with  it);  analysis  techniques  (how  you  understand  the  data);  and  collaboration 
tools  and  environments.  Collaboration  tools  enable  the  ability  to  connect  developers  and  users 
together  for  model  development,  scenario  development,  and  other  purposes. 

The  current  state-of-the  art:  I/O:  screens,  mice  and  keyboard  most  commonly  used; 
limited  use  of  audio;  few  haptic  devices;  close  coupling  of  computer  and  I/O  devices;  some 
research  available  on  human  perception  of  information;  desktop  and  windowing  paradigm  for 
presentation  which  is  predominantly  2D  graphics  with  increasing  use  of  3D. 
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Data  and  Information  Technology: 
-  Conservative  Predictions  -- 


•  Data  acquisition,  capture,  and  storage: 

-  Continues  to  be  fragmented,  ad  hoc  and  of  low  priority 

•  Transforming  data  into  useful  applications 

-  Some  automation  but  still  a  largely  manual,  time-consuming  process 

•  Presentation,  understanding,  collaboration: 

-  HQ:  Screens,  mice,  and  keyboards  replaced  by  voice  input,  light  pens,  electronic 
notepads; 

-  Collaboration  tools/environments  are  part  of  overall  system  design 


Data  acquisition,  capture  and  storage  continues  to  be  fragmented,  ad  hoc  and  of  low 
priority.  There  is  little  improvement  in  being  able  to  find,  access  and  understand  raw  data. 
Data  base  technology  is  not  the  limiting  factor,  desire  and  funding  is. 

Transforming  data  into  useful  applications,  like  scenario  generation,  sees  some 
automation  but  is  still  a  largely  manual,  time-consuming  process.  The  ability  to  take 
advantage  of  M&S  to  support  crisis  response  is  still  more  talk  than  action. 

Presentation,  understanding  and  collaboration  through  the  HCI  matures.  Screens,  mice 
and  keyboards  are  replaced  by  voice  input,  light  pens  and  electronic  notepads. 

Analysis  techniques  is  aided  by  user  interfaces  that  are  optimized  to  ease  analysis  of 
simulation  results.  The  ability  to  understand  simulation  outputs  is  still  an  art  versus  a  science. 

Collaboration  tools  and  environments  are  part  of  overall  system  design  and  real-time 
Video  Teleconferencing  (VTC)  and  whiteboarding  is  ubiquitous. 
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Data  and  Information  Technology 
-  Aggressive  Predictions  (1  of  2)  - 


•  Data  acquisition,  capture  and  storage 

-  More  systematic  instrumentation  of  experiments  and  events 

-  Database  and  data  warehouse  technologies  mature 

•  Transforming  data  into  useful  applications  (e.g.,  scenario  generation) 

-  Techniques  mature  to  automate  finding  relationships  in  raw  data 

-  Rapid  scenario  generation  technology  maturing 


TKoftc 


In  the  area  of  data  acquisition,  capture  and  storage  there  is  more  systematic 
instrumentation  of  selected  key  experiments  and  events.  The  storage  of  resulting  products  in 
data  warehouses.  Database  and  data  warehouse  technologies  mature  via  commercial  sector 
needs. 

Transforming  data  into  useful  applications  sees  techniques  mature  to  automate  finding 
relationships  in  raw  data.  Rapid  scenario  generation  technology  is  maturing  and  the  ability  to 
implement  within  next  generation  simulation  systems  is  realizable. 
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Data  and  Information  Technology 
-  Aggressive  Predictions  (2  of  2)  - 


•  Presentation,  understanding  and  collaboration: 

-  HCI:  Gesture  and  haptic  devices  mature 

-  Analysis:  Moving  towards  automatic  analysis;  focus  is  on  scenario  design,  not 
analysis  of  results 

-  Tools  mature  to  the  point  were  VTC  and  virtual  collaboration  is  the  norm 
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For  presentation,  understanding  and  collaboration  —  HCIs  include  mature  gesture  and 
haptic  devices. 

For  analysis  —  technology  is  moving  towards  automatic  analysis  with  the  focus  on 
scenario  design,  not  analysis  of  results.  Non- simulation  trained  users  are  able  to  become 
active  users  of  the  technology.  Tools  mature  to  the  point  were  VTC  and  virtual  collaboration 
is  the  norm.  This  is  commercially  driven  and  results  in  traveling  for  business  meetings  being 
down  25%. 
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Commercial  Technology  Trends  (1  of  4) 


•  Modeling  Methodology 

•  Internet  gaming  and  Internet  simulation 

•  General  purpose  computer  gaming 

•  Perception  modeling 

•  3-D  capture  devices  and  applications:  3-D  sound 

•  Interactive  multi-media:  Video  on  demand 

•  Situation  analysis 


Commercial  business  is  driving  the  listed  technologies.  It  is  likely  that  these  technologies 
will  have  value  to  DoD  M&S  developers  and  users.  It  is  unlikely  that  DoD  will  need  to  do 
more  than  use  or  participate  in  the  use  of  these  technologies.  DoD  project  offices  may  develop 
relationships  with  certain  vendors  to  extend  their  product  capabilities. 
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Commercial  Technology  Trends  (2  of  4) 


Development  Methodology 

•  Simulation  frameworks 

•  Intelligent  agents 

•  Automatic  programming 

•  Component  technologies 


Commercial  business  is  driving  the  listed  technologies.  It  is  likely  that  these  technologies 
will  have  value  to  DoD  M&S  developers  and  users.  It  is  unlikely  that  DoD  will  need  to  do 
more  than  use  or  participate  in  the  use  of  these  technologies.  DoD  project  offices  may  develop 
relationships  with  certain  vendors  to  extend  their  product  capabilities. 
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Commercial  Technology  Trends  (3  of  4) 

Computation  and  Communications 

•  Internet  (telecommunication/commerce) 

•  Personal/portable  computing  (appliance  computing 

•  Voice  recognition 

•  GPS  everywhere,  in  everything 

•  Video  on  Demand 

•  Flat  panel  displays/head  mounted  displays 

•  Very  low  cost,  high  speed,  high  resolution,  networked  computers  in  50% 
of  US  homes 

•  Long  life  batteries  (increased  portability) 


Commercial  business  is  driving  the  listed  technologies.  It  is  likely  that  these  technologies 
will  have  value  to  DoD  M&S  developers  and  users.  It  is  unlikely  that  DoD  will  need  to  do 
more  than  use  or  participate  in  the  use  of  these  technologies.  DoD  project  offices  may  develop 
relationships  with  certain  vendors  to  extend  their  product  capabilities. 
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Commercial  Technology  Trends  (4  of  4) 


Data  and  Information  Technology 

•  Knowledge  management 

-  Data  Mining  of  large  data  sets 

-  Nano-technology  (instrumentation/data  collection) 

-  Databases  and  utilization  tools 

-  Data  Visualization 

•  Virtual  meetings  (VTC,  collaboration,  chat  rooms) 

•  Planning,  scheduling  and  resource  allocation 

•  High  resolution,  low  cost  satellite  imagery 


gjjgg 


Commercial  business  is  driving  the  listed  technologies.  It  is  likely  that  these  technologies 
will  have  value  to  DoD  M&S  developers  and  users.  It  is  unlikely  that  DoD  will  need  to  do 
more  than  use  or  participate  in  the  use  of  these  technologies.  DoD  project  offices  may  develop 
relationships  with  certain  vendors  to  extend  their  product  capabilities. 
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Report  of  the 

Synthesis  Working  Group 


This  talk  is  a  short  set  of  musings  with  the  flavor  of  synthesis.  It  is  not  an  attempt  to 
summarize  the  summaries,  or  to  comprehensively  review  the  work  of  the  overall  workshop. 
Despite  this  caveat,  it  seems  that  there  is  a  pattern  to  what  is  in  the  air,  if  merely  we  listen  to  it 
correctly. 
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Participants 


Paul  Davis,  Co-Chair 
Clayton  Thomas,  FS,  Co-Chair 
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Something  Worthy  of  Being  Called  a  “Requirement” 

for  M&S  2007 


•  DoD’s  M&S  must  support: 

•  (1)  Adaptive  planning 

•  (2)  Commanders’  concerns  for  risk  and  uncertainty,  including 

-  That  related  to  smart  enemies  using  clever  and  perhaps  asymmetric  tactics 

-  And  that  related  to  complex  and  nonlinear  phenomena 

•  and 

•  (3)  Transforming  the  force  for  future  warfare  by  facilitating 
innovation,  testing,  and  refinement  of  operational  concepts  for  parallel, 
time-compressed,  network-centric  operations  dependent  less  on  classic 
head-on-head  attrition  battles  than  on  maneuver  of  disparate  capabilities 
(forces,  fires,  IW,  defenses,...)  to  destroy  or  otherwise  defeat  adversaries 
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The  word  “requirement”  is  often  used  in  disastrous  ways  in  large  organizations,  but  it  is  a 
useful  word  nonetheless.  This  slide  suggests  a  unifying  “requirement”  for  DoD’s  M&S.  The 
essence  is  that  if  M&S  is  to  be  useful,  it  must  support  —  and  not  merely  at  the  level  of  lip 
service  —  adaptive  planning,  assessment  of  risk  and  uncertainty  and  innovation. 
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With  Apologies  for  Using  the  “P”  Word,  This 
Requires  Some  Paradigm  Shifts 

•  Planning  for  adaptiveness  becomes  fundamental:  Planning  for  standard 
scenarios  becomes  old-think 

•  4 ‘Baseline  cases”  or  even  “best  estimates”  don’t  hack  it 

•  Modeling  dumb  enemies  —  or  superman  enemies  —  is  counterproductive 

•  Models  that  constrain  thinking  or  teach  rigidity  are  dysfunctional  and 
have  negative  value 

•  Models  should  perhaps  be  built  around  command  and  control,  networks 
and  systems  of  systems,  not  around  platforms  or  units 
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If  we  take  the  requirement  seriously,  it  has  many  implications.  This  slide  mentions  only 
some  of  them,  but  they  may  be  sufficient  to  demonstrate  that  the  implications  involve 
something  a  good  deal  more  than  changes  on  the  margin. 
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Themes  Suggested  for  the  Style  of  Analysis 

•  Exploratory  analysis  in  a  broad  scenario  space  (related  to  capabilities- 
based  planning) 

•  Examining  uncertainty  to  highlight  risks  to  be  reduced 

•  Exploiting  humans  for  both  creative  inputs  and  insightful  exploratory 
review 

•  Connecting  to  users,  spiral  analysis,. . . 

•  Distributed  collaboration  in  development,  analysis  and  operations 
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Suppose,  then,  that  we  accept  the  need  for  deep  changes.  What  kinds  of  analysis  will  we 
find  ourselves  doing  if  we  rise  to  the  challenges?  This  slide  mentions  some  of  the  key  themes. 
The  need  for  adaptiveness  and  the  ability  to  deal  with  risk  and  uncertainty  should  drive  us 
toward  exploratory  analysis  and  away  from  analysis  that  purports  to  be  accurately  predictive 
or  optimizing.  It  drives  us  toward  analysis  in  which  “probability”  distributions  are  sometimes 
prominent,  and  in  which  the  focus  is  on  understanding  “tails”  and  “blips,”  not  merely  alleged 
central  tendencies.  It  drives  us  toward  analysis  that  support  the  search  for  robustness, 
flexibility,  and  adaptiveness. 

Both  to  understand  uncertainties  and  to  better  understand  our  own  potentialities,  analysis 
must  actively  support  the  creative  process.  If  M&S  does  not  facilitate  creation  and  study  of 
innovative  operational  concepts,  then  it  may  be  counterproductive,  not  “merely”  dysfunctional. 
But  this  implies  —  if  experience  is  our  guide  —  that  M&S  must  actively  and  enthusiastically 
link  with  humans.  M&S  should  include  human  war  games  and  optionally  interactive 
simulations  (e.g.,  live  Blue  command-level  play  versus  smart  Red  models).  It  should  be 
flexible  enough  to  represent  the  full  range  of  strategies  and  tactics,  and  the  full  range  of 
important  factors  —  whether  or  not  they  lend  themselves  well  to  predictive  mathematics.  And 
they  should  lend  themselves  well  to  reflecting  insights  from  human  play  in  “smart”  models. 
This,  in  turn,  suggests  close  connectivity  between  modelers,  analysts  and  operators,  a 
connectivity  exploited  iteratively  —  in  a  “spiral”  process  —  rather  than  in  anything  more  neat. 
And,  finally,  it  suggests  great  emphasis  on  collaborative  methods  allowing  the  appropriate 
collaborations  and  competitions. 
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Enabling  Science  and  Technology 

•  Warfare  area  research  (precision  fires,  missile  defense,  littoral 
warfare,...;  transnational  threats,..critical  infrastructure...?) 

•  Basic  research  on  M&S  methodology,  environments,  tools,  and  standards 


If  we  have  a  sense  of  the  “requirement”  and  the  form  that  analysis  should  take,  what  about 
the  enablers?  This  slide  reminds  us  of  a  recurring  theme  of  the  workshop:  that  there  is  need 
not  only  for  “technology,”  but  also  for  an  enriched  base  of  military  science  and  an  enriched 
base  of  methodology. 

Drawing  in  part  on  work  reported  elsewhere  (National  Research  Council,  “Modeling  and 
Simulation,”  Volume  9  of  “Tactics  and  Technology  for  the  United  States  Navy  and  Marine 
Corps,  2000-3035,”  National  Academy  Press,  1997),  we  see  the  need  for  two  kinds  of 
research.  If  we  are  to  harvest  the  fruits  of  fundamental  research  (e.g.,  on  cognitive  science  or 
complexity  theory)  for  the  DoD’s  needs,  we  need  application-focused  research,  which  might 
be  organized  around  major  subjects  in  next-generation  military  operations.  Each  such  warfare 
area  has  its  own  phenomena,  which  are  much  more  complex  and  subtle  than  one  would  ever 
imagine  from  the  usual  viewgraph-level  discussions.  Each  such  area  deserves  a  research  base, 
but  one  that  might  take  10  years  to  develop  —  with  scientists  (not  just  programmers)  working 
closely  with  operators  and  with  those  developing  and  testing  new  concepts  and  systems. 
Creating  such  warfare-area  research  programs  could  be  a  relatively  inexpensive  but  powerful 
enabling  activity  for  the  DoD.  The  key  agencies  should  be  DARPA  and  DDR&E/DMSO. 

Although  there  is  a  great  deal  of  fundamental  science  that  begs  to  be  exploited  in  warfare- 
area  research  programs,  profound  problems  also  exist,  problems  that  are  simply  not  well 
understood  yet  and  that  deserve  many  years  of  study.  These  involve  concepts  of  M&S,  M&S 
environments  (including  optimal  use  of  interactiveness)  and  tools  generally.  We  should  not 
assume  that  the  commercial  world  will  provide  all  that  is  needed  here,  although  there  are  very 
nice  tools  that  have  emerged. 
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Subjects  for  Both  Fundamental  and  Applied 
Research  (1  of  2) 


•  Agent-based  modeling,  search  and  model  building 

•  Variable  structure  simulation 

•  MRM  (MultiResolution  Modeling),  families  of  models 

•  Cognitive  modeling  (representational  formalisms,  capturing  cognitive 
maps,  ...[not  how  brain  works]) 

•  Role  of  interactiveness  in  discovery,  definition,  analysis,  ...(man-machine 
partnership)  (HCI) 

•  Modeling  around  deep  knowledge  (declarative  plus  some  procedural) 
rather  than  algorithms 

•  Representation  for  and  reasoning  about  network-centric  systems 

•  Primers  on  M&S  for  domain  experts  and  analysts 

•  Distributed  learning;  collaborative  tools; 
digital  libraries;  cross  educating  for  training, 
analysis,... 


If  we  now  turn  from  generalities  to  “name  level”  subjects  that  seem  to  be  particularly 
relevant  to  our  image  of  analysis  needs,  we  see  the  items  in  this  slide.  If  there  are  to  be  major 
changes  of  the  sort  needed,  then  these  are  key  subjects  that  seem  to  offer  the  needed  types  of 
promise.  Agent-based  modeling  is  closely  related  to  models  of  both  decision  making  and 
behavior;  it  is  closely  related  to  giving  M&S  the  capacity  to  adapt  to  circumstances,  and  even 
to  reorganize  and  self-organize.  Variable-structure  simulation  is  not  well  studied  as  yet, 
although  some  books  on  the  subject  exist.* 

Agent  methods  are  also  needed  for  exploratory  analysis  —  to  help  make  sense  of  results 
in  huge  outcome  spaces  that  reflect  uncertainty.  They  can  help  with  inverse  questions  such  as 
“Under  what  circumstances  would  we  do  well?”  These,  rather  than  “What  if?”  questions, 
become  central.  And  agents  could  be  used  actually  to  build  “models,”  to  infer  from  simulation 
outcomes  and  empirical  work  with  human  players  what  factors  are  most  important  and  even 
what  cause-effect  processes  seem  to  dominate.  In  this  domain  and  in  others  as  well,  we 
believe  a  fundamental  element  of  success  will  be  interactiveness.  At  least  for  the  foreseeable 
future  we  believe  that  progress  will  be  most  feasible  where  the  man-machine  partnership  is 
developed,  where  model-building  or  outcome-exploring  agents  are  strongly  guided  by  insights, 
intuition,  and  flat-out  guesses  from  human  domain  experts.  This  is  drastically  different  from 
mere  statistical  analysis  as  currently  practiced,  but  also  very  different  from  the  vision  of 
computers  making  sense  of  chaos  by  themselves. 


*For  the  purposes  here,  “agents”  are  simulation  objects  that  serve  as  surrogates  for  or  agents  of 
humans.  One  kind  of  agent  is  a  decision  model  representing  a  commander.  A  second  kind  serves  the 
analyst  by  watching  for  states  to  arise  at  which  it  will  intervene  to  change  parameter  values  or  even 
structural  features  of  the  simulation  in  the  same  way  an  analyst  might  do  interactively  if  watching  state 
changes  continuously  on  some  imaginary  display.  As  noted,  many  different  functions  can  be  served  by 
agents.  Agents  are  implemented  in  many  ways.  Some  look  like  I/O  programs;  others  use  A. I.  methods 
such  as  branched  scripts  or  production  rules;  still  others  use  methods  sometimes  associated  with  artificial- 
life  studies  (e.g.,  cellular  automata).  These  implementation  details  are  less  important  here  than  the 
functions  served. 
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Subjects  for  both  fundamental  and  applied  research 

(2  of  2) 

•  Agent-based  modeling,  search  and  model  building 

•  Variable  structure  simulation 

•  MRM  (multi resolution  modeling),  families  of  models 

•  Cognitive  modeling  (representational  formalisms,  capturing  cognitive 
maps,  ...[not  how  brain  works]) 

•  Role  of  interactiveness  in  discovery,  definition,  analysis,  ...(man-machine 
partnership)  (HCI) 

•  Modeling  around  deep  knowledge  (declarative  plus  some  procedural) 
rather  than  algorithms 

•  Representation  for  and  reasoning  about  network-centric  systems 

•  Primers  on  M&S  for  domain  experts  and  analysts 

•  Distributed  learning;  collaborative  tools;  digital  libraries;  cross  educating 
for  training,  analysis.... 


So  also  multiresolution  modeling,  both  within  single  models  and  families  of  models,  is 
fundamental  to  a  rich  understanding  in  any  warfare  domain.  Moreover,  it  is  a  key  element  of 
dealing  with  complex  adaptive  systems  (such  as  warfare)  in  which  phenomena  may  have 
“laws  of  their  own”  at  different  levels. 

The  next  item  refers  to  focusing  human  efforts  on  the  expression  of  “deep  knowledge” 
rather  than  particulars  of  problem  solving.  It  is  far  more  powerful  to  express  knowledge  in  a 
form  like  Maxwell’s  Equations,  and  allowing  computers  to  find  solutions  for  particular 
situations,  than  to  have  humans  specify  for  the  computer  how  to  solve  each  new 
electromagnetics  problem.  So  also,  there  is  enormous  power  in  conservation  laws.  The 
analogs  in  the  military  domain  include  representing  objectives,  components,  and  constraints 
(e.g.,  mission  orders,  tactical  building-block  operations  and  constraints  imposed  either  by 
choice  or  the  situation).  Importantly,  this  modeling  around  deep  knowledge  rather  than 
algorithms  is  not  the  same  as  the  computer-science  distinction  between  declarative  and 
procedural  knowledge,  because  some  of  the  key  knowledge  is  in  fact  procedural.  This  is 
evident  in  real-world  operations,  which  depend  on  combining  building-block  operations  in 
particular  ways,  not  start-from-scratch  activities.  This  said,  one  of  the  most  challenging 
modeling  problems  is  how  to  represent  and  deal  with  “network”  systems  where  important 
properties  of  the  network  are  “emergent”  and  not  well  understood.  This  is  becoming  a  crucial 
issue  as  US  military  operations  become  more  network  centric  and  more  dependent  on  M&S 
embedded  in  the  command  and  control  system. 

If  man-machine  partnerships  are  essential,  as  they  are  in  command  and  control,  then 
collaborative  work  can  be  enriching  and  even  essential.  This,  in  turn,  will  depend  upon  the 
ability  to  share  information,  models  and  analyses  easily.  The  methods  currently  being  used  for 
lifelong  learning  and  distributing  learning  seem  highly  relevant  here,  especially  if  the  future 
will  include  our  wanting  quickly  to  be  able  to  comprehend  and  try  out  models  that  others 
somewhere  in  cyberspace  have  developed. 


Appendix  B.5  -  8 


Stake-Driving  Needed 


Central  Planning 
Bureaucratic  W&A 
Corruption  for  advocacy 
Consensus  and  consistency 
Ober  alles 

“Objectivity”  worship  (as  in 
“no-soft-factors  here!”) 


The  last  two  slides  are  somewhat  different  in  character,  but  are  again  efforts  to  capture 
key  ideas  floating  through  the  workshop.  This  slide  notes  that  current  M&S-supported 
analysis  is  severely  hampered  by  what  many  see  as  DoD’s  tendencies  —  especially  in  the  last 
ten  years  —  to  approach  M&S  matters  with  excessive  controls,  bureaucracies,  processes  and 
procedures  —  many  of  which  seem  to  serve  the  needs  of  advocacy  or  don’t-rock-the-boat 
thinking  than  the  needs  of  innovation  or  the  search  for  insight.  This  point  is  becoming  more 
widely  recognized  by  senior  people  in  the  community,  including  decision  makers.  Related 
discussion  can  be  found  in  the  1996  DSB  summer  study  report  (Vol.  2),  in  the  opening  talk  to 
the  workshop  by  General  Larry  Welch  (USAF,  retired),  and  in  many  other  documents.  What 
is  new  is  that  more  senior  figures  appear  to  be  aware  of  the  problems. 
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Problem:  how  to  control  control 
(helping,  not  hurting) 


Desirable 

Undesirable 

Enabling  standards  (MIME, 
...HLA) 

"Blessed11  models,  data 
bases,  stamping  out 
"redundancy" 

Subsidizing  infrastructure 
creation  (with  broad 
architecture ) 

Specifying  detailed 
character  of  infrastructure 

Base  cases  for  comparison 

Treating  base  cases  as 
meaningful  best  estimates 

Infrastructure  for 
distributed  collaboration 

"Blessed"  federations 

Connecting  modelers  with 
operators  and  other  users 

Laundry-list  "requirements" 
from  operators  and  other 
users 

Requirement  to  treat  soft 
factors 

Requirement  to  model 
cognition,  physiology... 

Finally,  this  slide  suggests  some  important  distinctions  that  may  help  in  attempting  o 
move  forward.  These  are,  for  example,  the  distinction  between  having  enabling  standards 
(e.g.,  the  MIME  standard  making  cross-platform  e-mail  possible,  or  the  HLA)  versus  having 
shackles  such  as  insistence  on  the  use  of  bureaucratically  “blessed”  models  and  data  bases, 
whether  or  not  appropriate,  or  having  active  efforts  to  stamp  out  alleged  redundancy,  which 
has  the  effect  of  stamping  out  the  competition  of  ideas  so  necessary  for  progress. 

The  other  examples  can  be  read  as  they  stand.  The  bottom  line  here  is  that  we  must  all 
recognize  that  many  of  the  initiatives  that  could  be  of  greatest  value  to  the  community  and  the 
policymakers  we  serve  bring  with  them  “dark-side”  features  that,  if  not  controlled,  can  make 
things  worse  rather  than  better.  At  the  same  time,  being  against  the  initiatives  because  of  the 
potential  bad  actions  will  doom  the  community  to  increasing  irrelevance  and  incompetence. 
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What  Was  Said  (11/88):  SIMTECH  97  Findings 

Tasking  sought  to  use  technology  to  solve  a  range  of  life  cycle 
management  issues 

Working  Group  concluded  that  technology  was  a  tertiary  issue,  that  by 
itself 

-  Doesn't  solve  many  existing  problems 

-  Will  not  provide  desired  enhancements 

Instead,  community  needs  to  address  management  and  implementation 
issues 

-  Need  to  resolve  these  issues  to  take  advantage  of  technology 

-  Need  a  more  appropriate  life  cycle  model 


Appendix  C.  1  -  3 


Alternative  Life  Cycle  Views 
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Elements  of  the  Life  Cycle 
Discussed  in  SIMTECH  97  (11/88) 


•  Human  capital 

-  Development  skills  (functional  and  software) 

-  Analysis  skills 

•  Military  knowledge  base 

-  Basic  research 

-  Aggregation/abstraction 

•  Sponsor  interactions 

-  Substantive  support 

-  Appropriate  expectations 

•  Implementation  environment 

-  Hardware 

-  Software 
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What  Did  SIMTECH  97  Do  Well 
On  Life  Cycle  Management? 

•  Technology  didn’t  and  won’t  solve  many  life  cycle  management  issues; 

M&S  management/implementation  issues  dominated 

-  Identified  many  of  the  key  issues  that  impaired  the  capabilities  of  military 
simulations 

-  Some  have  been  partially  addressed  since  SIMTECH  97  (e.g.,  scenario 
development  software,  DMSO, ...) 

-  Some  issues  remain  as  problems  today 

•  Assessment  that  the  role  of  domain  experts  is  key 

-  But  DoD  does  a  poor  job  of  training,  supporting  and  using  these  people  in  M&S 
development 

•  Recognized  the  importance  of  using  commercial  products 

-  Example,  need  for  automated  development  tools,  data  input  and  collection  tools 

-  But  the  trend  is  even  stronger  now  (did  not  foresee  breadth  of  technical 
breakthroughs,  such  as  virtual  immersion) 
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Some  Shortfalls  of  SIMTECH  97 


•  MORS  community  failed  to  adequately  market  the  life  cycle  management 
recommendations  and  achieve  their  implementation 

-  Need  to  employ  a  proponent  group  to  continually  press  key  SIMTECH  2007 
recommendations  (ongoing  briefings,  staff  interactions, ...) 

•  SIMTECH  97  failed  to  project  changes  in  the  environment  (end  of  Cold 
War,  changes  in  technology,  reduced  funding  and  manpower, ...);  not 
even  considered 

-  Need  to  be  more  explicit  of  expected  changes  in  SIMTECH  2007  (including  RMA, 
commercial  software  evolution,  pace  of  technology  change,  geopolitical,  cultural, 
social,  economic ...) 

-  Need  to  document  assumptions,  cover  the  range  of  key  uncertainties 

•  Did  not  see  extent  that  commercial  companies  would  leverage  DoD 
software  investments,  provide  superior  environments  for  future  DoD 
work  (spin  off/spin  on) 


Egg 
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Other  Observations  for  the  Future 


-  Does  corporate  management  of  DoD  simulations  need  to  be  integrated  (not  by 
service)? 

-  Military  acquisition  process  lacks  flexibility  for  military  simulation  acquisition 

•  Need  to  fix  requirements  generation  process  (need  to  consider  what  is  doable,  but  then  let 
technologists  push  the  envelope) 

-  Future  models  need  to  be  designed  to  accommodate  requirements  and  technology 
changes 

-  Many  models  will  be  systems  of  models 

•  Models  need  to  be  managed  like  C4ISR  systems  where  continuous  change  is  expected 

-  How  far  should  we  look  ahead?  —  10  years  max 
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SIMTECH97  Recommendations 


•  Workbench  architecture 

-  Enunciate  architecture 

-  Develop  common  specification  language 

-  Continue  development  of  simulation  languages 

-  Develop  ties  between  specification  and  simulation  languages 

-  Determine  workbench  security  requirements 

•  Define  knowledge  base  for  military  simulations  (Common  iconography, 
OPLANs,  algorithms,  data  standards,  data  bases,  M&S,  M&S  W&A 
documents,  data  W&C  documents,  etc.) 

•  Development,  management  and  use  of  repository 

•  Research  in 

-  Spatial-temporal  reasoning 

-  M&S  performance  enhancements 

•  Use  of  Parallelism 

•  Special  hardware 

•  Object  oriented:  CPU  architecture,  software  development 


The  WG  defined  an  analyst  workbench  as  an  evolving  architecture  providing  an  integrated 
collection  of  standards,  protocols  and  tools  to  enable  analysts  (including  researchers  and  users)  to  build, 
understand,  use  and  reuse  simulations,  using  a  range  of  approaches  methodologies  and  languages. 

Three  key  needs  were  to  be  satisfied  in  order  to  develop  and  maintain  such  a  capability.  They  included: 
an  extensible,  readable  standard  specification  language  for  describing  models;  an  extensible  semantic 
dictionary  to  provide  a  sharable  repository  of  models,  submodels  and  data;  and  an  architecture  for 
integrating  the  tools  and  the  repository  by  identifying  components  and  specifying  what  must  be 
standardized. 

The  WG  identified  five  broad  technology  initiatives  that  should  be  undertaken  as  a  foundation  for 
the  analyst  workbench.  These  were:  (1)  develop  the  architecture  (e.g.,  enunciate  the  architecture, 
develop  a  common  specification  language,  continue  development  of  simulation  languages,  develop  ties 
between  specification  and  simulation  languages,  determine  workbench  security  requirements);  (2) 
assemble  the  knowledge  base  for  military  simulation  (e.g.,  common  iconography,  operational  plans); 

(3)  perform  research  in  spatial-temporal  reasoning;  (4)  perform  research  to  achieve  key  performance 
enhancements  (e.g.,  parallelism,  object-oriented  CPU  architecture);  and,  (5)  assemble  repository 
elements  (e.g.,  survey  tools,  find/prioritize  software). 

The  WG  further  recommended  that  the  community  should  monitor  but  not  fund  efforts  in  voice 
recognition,  graphics  or  natural  languages,  since  it  believed  that  adequate  work  was  in  progress  in  the 
commercial  sector. 
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What  Did  SIMTECH97  Do  Well? 


•  Identified  need  for 

-  Common  workbench  infrastructure,  standards  and  tools 

-  Knowledge  base  for  military  simulations 

-  Development,  management  and  use  of  an  M&S  repository  system 

-  Workbench  security  requirements 

-  Research  in  object-oriented  development 

-  Research  into  spatial -temporal  reasoning 


Egg 


The  WG  did  a  good  job  in  identifying  the  main  components  necessary  to  support  an  analyst 
workbench.  First  was  the  common  workstation  infrastructure,  standards  and  tools  that  would  provide  the 
basis  for  information  exchange,  interoperability,  portability,  reuse  of  M&S  components,  etc.  Next,  the 
objects  and  products  to  be  used  in  the  workbench  needed  to  be  identified  and  made  available  in  a 
knowledge  base  for  military  simulations.  These  included:  standards  for  data,  symbols,  icons, 
nomenclature,  protocols;  M&S  and  their  components;  databases;  battlefield  functions;  operational  plans; 
tools/toolkits;  etc.  Next  was  the  repository  system  that  registered,  managed,  searched,  validated  and  in 
general  made  available  the  knowledge  base  products. 

The  WG  realized  that  a  major  issue  would  be  the  security  classification  of  the  workbench  which 
preferably  would  be  multilevel  but  would  at  least  need  to  be  able  to  operate  at  different  system  high 
levels  for  different  uses.  Even  more  important  was  support  for  knowledge  base  products  at  different 
security  levels  and  how  the  analyst  could  bring  them  together  on  the  workbench  to  address  the  problem 
at  hand.  Practical  solutions  to  the  security  problems  are  still  needed. 

Lastly  research  continues  in  two  areas  identified  in  1987:  object-oriented  development  and  spatial- 
temporal  reasoning.  The  1987  focus  was  primarily  on  object-oriented  CPU  architecture  whereas  the 
focus  in  1997  is  on  object-oriented  software  life  cycle  development  (e.g.,  the  Unified  Modeling 
Language  (UML)),  object-oriented  information  modeling  (e.g.,  IDEF1X97),  object-oriented  distributed 
systems  (e.g,  CORB  A),  all  of  which  are  critical  to  M&S.  The  two  major  new  DoD  modeling  efforts, 
JWARS  and  JSIMS,  are  using  object-oriented  software  approaches  for  design  and  implementation. 

The  research  in  spatial-temporal  reasoning  is  still  critical  to  M&S  and  has  not  moved  as  quickly  as 
was  anticipated  in  1987.  Recent  ACTD  and  ATD  efforts  have  been  addressing  the  rapid  provision  of 
terrain  data  on  demand  which  can  serve  as  a  basis  for  advances  in  spatial-temporal  reasoning. 
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Type  I  Errors 


•  Failure  to  foresee: 

-  Major  technology  advances 

•  Rapid  advances  in  chips,  computers,  memories,  workstations,  desktop  computing,  laptops, 
networks,  communications 

•  Development  of  internet  and  web,  Java  language,  sharing  of  IT  products 

•  Development  of  collaborative  software 

-  Cold  WarrEnding  resulting  in  reduced  resources  for  analysis 

-  Training:  Capturing  the  playing  field  and  M&S  resources  and  raising  the  demand 
for  interoperable  M&S  and  development  of  DIS  and  HLA 

-  Advances  in  standards:  C4ISR  community  emphasis  on  interoperability  of 
information  systems  resulting  in  TAFIM,  TRM,  JTA,  DII  DOE,  data  standards  — 
all  applicable  to  information  infrastructure  for  analyst  workbench 

-  Languages:  DoD  rescinding  the  Ada  mandate 

-  Tools:  Uncertainty  and  complexity  becoming  major  analysis  concerns  and 
requiring  new  tools  and  approaches 


yj|gjg 


As  can  be  seen,  the  WG  did  not  do  a  very  good  job  in  predicting  what  the  environment  would  look 
like  ten  years  hence.  To  begin  with,  the  very  rapid  advances  in  computer  and  communications 
technologies  were  not  anticipated  as  was  the  commercialization  of  the  ARPANET  into  the  Internet  and 
the  rapid  expansion  of  the  web  and  more  recently  the  Java  language.  Much  of  the  surprise  has  been  due 
to  the  willingness  of  the  commercial  world  to  share  information  technology  products  even  before  the 
infrastructure  to  support  quality,  protection,  intellectual  rights,  etc.  has  been  developed  and  policy  put 
in  place. 

The  end  of  the  cold  war  was  not  anticipated  as  was  the  reduction  in  DoD  funding  and  the  lack  of 
resources  for  analysis  at  a  time  when  the  uses  of  analysis  have  been  rapidly  expanding.  For  example, 
DoD  would  like  to  reduce  costs  by  using  M&S  as  an  integral  part  of  the  acquisition  process  as  well  as 
in  training  and  exercises.  Indeed  the  concept  of  distributed  simulation  which  was  fairly  new  with 
SIMNET  technology  in  1987  grew  more  rapidly  than  anticipated  when  the  training  community  realized 
that  live,  virtual  and  constructive  M&S  could  be  used  together  in  training  exercises.  SIMNET  led  to 
Distributed  Interactive  Simulation  (DIS)  which  is  based  mainly  on  Protocol  Data  Unit  (PDU)  standards 
and  that,  in  turn,  has  led  to  DMSO’s  definition  of  the  High  Level  Architecture  (HLA)  that  supports 
simulations  interacting  together  as  a  federation.  HLA  includes  rules,  specification  of  a  Run  Time 
Infrastructure  (RTI),  the  Application  Programming  Interface  (API)  specifications  that  M&S  need  to 
interface  with  the  RTI,  the  information  about  the  federates  used  by  the  RTI  during  runtime  that  is 
captured  in  a  Federation  Object  Model  (FOM). 

Another  unanticipated  happening  was  the  massive  push  in  the  C4ISR  community  to  really  tackle 
the  lack  of  interoperability  among  C4ISR  systems  and  more  recently  C4ISR  systems,  weapon  systems, 
sustaining  base  systems  and  M&S.  The  Joint  Technical  Architecture  (JTA)  describes  the  standards 
necessary  for  IT  systems  to  interoperate  in  several  areas:  processing  standards,  transfer  standards, 
information  modeling  and  message  standards,  Human  Computer  Interface  (HCI)  standards  and  security 
standards.  The  use  of  common  software  to  form  the  basic  infrastructure  for  a  C4ISR  workstation 
platform  that  is  interoperable,  portable,  scalable  is  defined  by  the  Defense  Information  Infrastructure 
(DII)  Common  Operating  Environment  (COE). 

The  1987  WG  did  not  anticipate,  but  did  hope,  that  DoD  would  rescind  the  Ada  language  mandate. 

Lastly,  the  WG  felt  that  the  1987  WG  was  lacking  in  not  paying  more  attention  to  the  analytic  need 
to  address  uncertainty  and  complexity  in  military  simulations. 
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Type  II  Errors 


•  Wrongly  predicted  a  need  for  M&S  special  purpose  hardware/software 
architectures  and  parallel  architectures 


During  the  1980s  there  was  much  R&D  activity  focused  on  parallel  architectures  and  special 
purpose  architectures  to  achieve  performance  improvements  in  M&S.  Parallelism  at  the  fine  grained 
level  within  processors  has  overwhelmed  the  coarse  grained  parallelism  that  was  thought  to  be  so 
important  when  it  was  predicted  that  the  industry  would  be  bumping  into  technology  limits.  Within  a 
few  years  we  will  have  1GHz  processors  with  even  more  pipelining  and  superscalar  pipelines,  SIMD, 
etc.  built  in.  So  parallelism  turned  out  to  be  more  useful  for  distribution  than  for  performance  reasons. 

The  rapid  and  continuing  increase  in  performance  of  "standard"  commercial  chips,  CPUs,  graphics 
engines,  workstations  along  with  reduction  in  size  and  cost  makes  special  purpose  solutions  too  costly 
and  unnecessary.  The  use  of  COTS  IT  will  yield  increasing  performance  gain  through  evolutionary 
product  development  with  business  incentives  to  be  backward  compatible. 
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Assessment  of  SIMTECH97 
Moving  into  Today  (1  of  4) 


Workbench  architecture  infrastructure  being  addressed  by  DoD  M&S 
Master  Plan  and  the  C4ISR  community  interoperability  efforts 
M&S  common  specification  language  as  a  way  to  capture  M&S 
functionality,  being  partially  met  by  HLA  SOM  and  FOM,  VV  &  A 
documentation 

M&S  implementation  language  need  affected  by  DOD  rescinding  Ada 
mandate 


Knowledge  base  for  military  applications  being  worked  on  by  M&S 
community 

Repository  being  addressed  by  DMSO  establishment  of  MSRR 
OO:  commercial  developments  and  use  in  JWARS  and  JSIMS 
Spatial-temporal  reasoning:  environment  EAs,  continued  research  needed 
Areas  not  being  sufficiently  addressed:  security;  HCI  for  workbench; 
standards  for  data,  symbology,  iconology,  nomenclature,  battlefield 
functions,  algorithms;  authoritative  sources  for  models;  reuse  of  M&S 
components 


Workbench  architecture  infrastructure  needs  are  currently  being  addressed  by  the  M&S  community 
through  implementation  of  the  DoD  M&S  Master  Plan  being  carried  out  by  the  DMSO,  services  and 
agencies  and  by  the  C4ISR  community’s  push  toward  interoperability  of  systems  through  the  use  of 
interface  standards  and  specifications  and  common  use  of  infrastructure  products.  DoD  has  mandated 
the  use  of  the  HLA,  JTA,  DII  COE  in  all  DoD  information  systems  (including  M&S  systems).  The 
DoD  M&S  Master  Plan  objectives  include  establishment  of  an  M&S  framework  (HLA,  Conceptual 
Models  of  the  Mission  Space  (CMMS),  data  standards);  authoritative  representations  for  environment, 
systems  and  human  behavior;  and  an  infrastructure  that  includes  M&S  W&A  documentation  and  the 
MSRR  —  all  of  which  are  critical  to  the  workbench  architecture.  The  C4ISR  community  has 
developed  the  DoD  Technical  Architecture  for  Information  Management  (TAFIM)  from  which  the 
Technical  Reference  Model  (TRM)  has  been  extensively  used  to  understand  the  critical  interface  areas 
for  interoperability  among  C4ISR  platforms.  The  JTA  is  essentially  a  DoD- wide  profile  of  consensus- 
based  standards  and  specifications  to  enable  interoperability  among  workstation  platforms.  The  current 
DII  COE  specifies  software  products  to  support  three  platforms:  Sun,  HP,  WINTEL  and  defines  the 
APIs  to  common  products  to  be  shared  across  DoD  agencies  and  services  such  as  the  Joint  Mapping 
Toolkit.  These  DII  COE  software  products  are  the  infrastructure  for  the  workbench.  M&S  domain 
specific  software  (such  as  the  RTI)  and  tools  will  become  a  part  of  the  shared  M&S  domain 
environment. 

An  M&S  common  specification  language  was  recommended  in  1987  as  a  way  to  capture  M&S 
information  and  functionality  so  that  an  AI  system  could  help  an  analyst  find  relevant  M&S  or  M&S 
components  for  possible  reuse.  Though  this  hasn’t  happened,  the  Simulation  Object  Model  (SOM)  and 
the  Federation  Object  Model  (FOM)  required  by  the  HLA  to  capture  information  about  the  simulation 
or  federation  as  to  the  objects  to  be  shared,  time  management,  etc.  captures  much  of  this  information  in 
a  standard  format  that  can  be  manipulated  by  software  (such  as  the  RTI).  In  addition,  M&S  W&A 
documentation  registered  with  the  MSRR  and  accessed  through  it,  contains  descriptive  information 
about  the  M&S,  its  purpose,  scenarios  it  can  support,  previous  uses,  data  and  information  sources,  etc. 
The  current  WG  is  unaware  of  efforts  to  develop  a  general  M&S  specification  language  but  there  are 
some  efforts  in  that  direction  in  the  software  industry  (e.g.,  UML)  and  a  few  specific  simulation 
specification  languages  such  as  the  Condition  Specific  Language  (CSL).  The  Air  Force  JMASS 
program  has  also  developed  a  language  for  specifying  JMASS  models.  The  WG  suggests  re-examining 
this  need  for  2007. 
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Assessment  of  SIMTECH97 
Moving  into  Today  (2  of  4) 


•  Workbench  architecture  infrastructure  being  addressed  by  DoD  M&S 
Master  Plan  and  the  C4ISR  community  interoperability  efforts 

•  M&S  common  specification  language  as  a  way  to  capture  M&S 
functionality,  being  partially  met  by  HLA  SOM  and  FOM,  W  &  A 
documentation 

•  M&S  implementation  language  need  affected  by  DOD  rescinding  Ada 
mandate 

•  Knowledge  base  for  military  applications  being  worked  on  by  M&S 
community 

•  Repository  being  addressed  by  DMSO  establishment  of  MSRR 

•  00:  commercial  developments  and  use  in  JWARS  and  JSIMS 

•  Spatial-temporal  reasoning:  environment  EAs,  continued  research  needed 

•  Areas  not  being  sufficiently  addressed:  security;  HCI  for  workbench; 

standards  for  data,  symbology,  iconology,  nomenclature,  battlefield 
functions,  algorithms;  authoritative  sources  for  models;  reuse  of  M&S 
components  (Egg 


The  development  of  an  M&S  implementation  language  suggested  in  1987  was  largely  a  reaction 
against  the  DoD  mandate  that  Ada  be  the  only  programming  language  used  in  DoD  information 
systems.  Recently,  DoD  has  rescinded  this  mandate  and  the  latest  JTA  Version  2.0  draft  says 
"programming  language  selections  should  be  made  in  the  context  of  the  system  and  software 
engineering  factors  that  influence  the  overall  life-cycle  costs,  risks,  potential  for  interoperability."  The 
WG  noted  that  different  domains  have  developed  high  level  languages  for  specific  purposes  that  are 
compilable  into  C  or  C++. 


Appendix  C.2  -  8 


Assessment  of  SIMTECH97 
Moving  into  Today  (3  of  4) 


•  Spatial-temporal  reasoning:  environment  EAs,  continued  research  needed 

•  Areas  not  being  sufficiently  addressed:  security;  HCI  for  workbench; 
standards  for  data,  symbology,  iconology,  nomenclature,  battlefield 
functions,  algorithms;  authoritative  sources  for  models;  reuse  of  M&S 
components 


The  need  for  a  knowledge  base  for  military  applications  is  being  partially  met  by  execution  of  the 
DoD  M&S  Master  Plan  that  requires:  authoritative  representations  for  environment,  systems  and  human 
behavior;  authoritative  data  and  model  sources;  Executive  Agents  (EA)  for  environment;  workshops  for 
human  behavior  models/standards;  push  for  M&S  data  models  and  standards  for  Data  Interchange 
Formats  (DIFs);  SOMs  and  FOMs;  CMMS;  and  W&A  documentation  guidelines.  The  Army  M&S 
Office  (AMSO)  also  has  an  effort  to  define  standards  including  those  for  algorithms  and  battlefield 
functions.  DISA  is  responsible  for  the  DoD- wide  data  standardization  effort.  M&S  data  standards 
should  be  nominated  and  incorporated  into  the  Defense  Data  Model  and  Data  Dictionary  to  provide 
interoperable  standards  across  C4ISR  and  M&S.  More  effort  is  needed  to  define  the  types  of  objects 
needed  in  the  knowledge  base,  some  means  of  standardizing  them,  and  ways  to  register,  validate, 
maintain,  search  for  them. 

The  DMSO  establishment  of  web-based  MSRRs,  one  unclassified  and  one  at  the  secret  level, 
directly  support  the  workbench  need  for  a  repository.  DMSO  also  provides  organizational  support  for 
managing  the  MSRRs  and  a  joint  working  group  to  guide  their  development.  The  WG  recognizes  a 
need  for  a  language  in  which  to  tailor  agents  to  search  for  information  products  in  the  MSRR. 

Relevant  commercial  activity  exists  in  three  Object-Oriented  (OO)  areas:  OO  software  life  cycle 
development  (e.g.,  the  UML),  OO  information  modeling  (e.g.,  IDEF1X97)  and  OO  distributed  systems 
(e.g,  CORBA).  Two  major  new  DoD  modeling  efforts,  JWARS  and  JSEMS,  are  using  OO  software 
approaches  for  design  and  implementation.  The  WG  recognizes  a  need  for  a  high  level  M&S  object 
taxonomy  to  ensure  some  compatibility  across  OO  M&S. 
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Assessment  of  SIMTECH97 
Moving  into  Today  (4  of  4) 


*  Spatial-temporal  reasoning:  environment  EAs,  continued  research  needed 

•  Areas  not  being  sufficiently  addressed:  security;  HCI  for  workbench; 
standards  for  data,  symbology,  iconology,  nomenclature,  battlefield 
functions,  algorithms;  authoritative  sources  for  models;  reuse  of  M&S 
components 


(Egg 


In  the  area  of  spatial-temporal  reasoning,  the  recent  DoD  vision  calling  for  support  for  precision 
weapons  and  information  dominance  has  led  to  new  requirements  for  higher  resolution  elevation  data 
and  features.  The  National  Image  and  Mapping  Agency  (NIMA)  was  formed  from  several  separate 
DoD  organizations  and  through  ATDs  and  ACTDs  is  addressing  needs  for:  high  resolution  terrain, 
terrain  elevation  and  feature  data  on  demand  with  timely  delivery,  support  for  handling  temporary  and 
long  term  battlefield  changes  in  terrain  ("dynamic  terrain"),  the  use  of  commercial  terrain  products. 
DMSO  has  established  Executive  Agents  for  the  environmental  areas.  The  Army  and  DMSO  are 
supporting  the  Synthetic  Environment  Data  Representation  and  Interchange  Specification  (SEDRIS)  to 
address  sharing  and  reuse  of  environmental  data  among  M&S.  The  intelligence  community  has 
continuing  efforts  to  fuse  and  integrate  data  from  multiple  sources.  Limited  papers  have  been  written 
that  address  temporal  logic  for  M&S  but  much  research  still  remains  to  be  done  in  spatial-temporal 
reasoning. 

Lastly  areas  that  have  not  received  sufficient  attention  are:  security;  Human  Computer  Interface 
(HCI)  for  the  workbench;  standards  for  data,  symbology,  iconology,  nomenclature,  battlefield 
functions,  algorithms;  authoritative  sources  for  models;  reuse  of  M&S  components. 
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Recommendations  (1  of  2) 


•  Re-examine  requirements  for  analyst  workbench 

•  Re-examine  need  for  general  M&S  specification  language  and/or 
implementation  language 

•  Recommend  defining  areas  that  the  M&S  tools/capabilities  should 
support,  specifically:  C4ISR,  uncertainties,  W&A,  extending  range  of 
OR  and  expert  tools 

•  Recommend  efforts  in: 

-  Security  solutions 

-  HCI  for  workbench 

-  Knowledge  base  for  military  applications 

-  MSRR,  including  language  to  tailor  agent  for  searching  MSRR 

-  Research  in  spatial-temporal  reasoning 

-  Common  M&S  OO  taxonomy 


The  WG  recommends  much  more  attention  be  paid  to  the  users  of  the  workbench,  their  needs, 
workbench  priorities  with  respect  to  providing  support  for  various  parts  of  the  M&S  life  cycle.  The 
WG  needs  to  consider  what  analyses  roles  the  workbench  should  support.  Should  the  tools  support 
development  of  M&S  by  software  developers,  and/or  development  of  M&S  by  reusing  components 
through  a  high  level  interface,  perhaps  interactively  by  decision  makers  or  staffers?  Should  the 
workbench  be  designed  for  analysts  who  have  OR  type  of  expertise  or  should  it  also  be  made  available 
to  decision  makers  and/or  staffers?  If  used  by  decision  makers  and/or  staffers,  then  how  can  they  be 
helped  to  understand  the  implications  of  their  choices,  how  will  they  or  should  they  perform  W&A. 

The  WG  recommends  that  the  needs  for  a  common  M&S  specification  language  and  one  or  more 
implementation  languages  be  re-examined.  The  S1997  recommendation  to  develop  an  M&S 
specification  language  was  primarily  to  enable  software  assistance  to  the  analyst  in  finding  the  M&S, 
components,  etc.  that  were  relevant.  The  HLA  requirement  that  each  M&S  have  a  SOM  that  defines 
specific  M&S  characteristics  relevant  to  an  HLA  federation  and  for  a  federation  to  define  a  FOM  to  be 
used  by  the  RTI  partly  answers  the  need  recognized  in  1987.  M&S  W&A  documentation  will  also 
provide  textual  information  about  the  respective  M&S  including  its  purpose,  scenarios,  data,  previous 
uses,  etc.  In  addition,  the  WG  recommends  that  an  analytic  object  model  be  defined  to  extend  the  SOM 
to  include  additional  types  of  M&S  information  specific  to  analysis. 

The  WG  was  unsure  as  to  whether  the  software  community  at  large  has  developed  a  generalized 
specification  language,  but  believes  the  UML  methodology  may  contain  one.  Several  people  were 
aware  of  specialized  M&S  specification  languages,  such  as  one  for  models  participating  in  the  AF 
JMASS  program. 

With  respect  to  an  M&S  implementation  language,  the  SIMTECH97  issue  was  that  a  common 
M&S  implementation  language  would  be  more  appropriate  for  M&S  development  than  Ada  which  at 
that  time  was  mandated  by  DoD.  The  Ada  mandate  was  recently  rescinded  and  so  there  may  no  longer 
be  a  need  for  a  common  M&S  implementation  language. 
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Recommendations  (2  OF  2) 


•  Re-examine  requirements  for  analyst  workbench 

•  Re-examine  need  for  general  M&S  specification  language  and/or 
implementation  language 

•  Recommend  defining  areas  that  the  M&S  tools/capabilities  should 
support,  specifically:  C4ISR,  uncertainties,  W& A,  extending  range  of 
OR  and  expert  tools 

•  Recommend  efforts  in: 

-  Security  solutions 

-  HC1  for  workbench 

-  Knowledge  base  for  military  applications 

-  MSRR,  including  language  to  tailor  agent  for  searching  MSRR 

-  Research  in  spatial-temporal  reasoning 

-  Common  M&S  OO  taxonomy 


.The  WG  recommends  more  attention  be  paid  to  determine  the  areas  that  the  workbench  tools 
should  support  such  as:  C4ISR  modeling  which  is  inadequately  done  in  most  M&S  and  is  especially 
necessary  as  the  military  continue  to  focus  on  information  dominance;  representation  and  handling  of 
uncertainties  and  complexity;  M&S  of  W&A;  and  extending  the  range  of  OR  and  expert  tools.  The 
WG  recommends  continuing  efforts  in  the  areas  of:  security,  HCI,  knowledge  base  (including  standards 
for  data,  symbology,  iconology,  nomenclature,  battlefield  functions,  algorithms;  authoritative 
representations  for  models);  MSRR  especially  with  respect  to  reuse  of  M&S  components;  research  in 
spatial-temporal  reasoning;  and  development  of  a  common  M&S  OO  taxonomy.  The  security  area 
needs  attention  with  respect  to  (1)  a  workbench  alternative  to  multilevel  if  that  is  unobtainable,  (2) 
attention  to  integrating  information  objects  across  classification  levels,  (3)  handling  a  change  in 
classification  level  due  to  aggregation  of  large  amounts  of  information,  (4)  the  need  within  the  M&S 
community  for  DoD  policy  that  supports  and  incentivizes  sharing  of  M&S  and  data.  The  WG 
recognizes  a  need  for  a  workbench  HCI  and  style  guide  that  would  be  compliant  with  the  JTA  but 
would  be  tailored  to  meet  the  specific  needs  of  the  M&S  domain.  The  WG  recommends  continued 
development  of  the  knowledge  base  and  the  MSRR  capabilities  to  support  reuse  of  M&S  components 
and  continued  research  on  spatial-temporal  reasoning.  Lastly,  WG  recommends  that  the  M&S 
community  develop  a  high  level  common  OO  taxonomy  to  support  the  development  of  consistent  and 
integrable  object  hierarchies  across  M&S. 
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Lessons  Learned  from  SIMTECH97 


•  SIMTECH97  didn’t  pay  enough  attention  to  predictions  of  future 
environment:  perhaps  get  brief  from  futurist  as  common  base  for  next 
workshop 

•  SIMTECH97  WG  didn’t  address  who  the  analyst  is/are,  the 
needs/requirements  for  the  workbench,  the  workbench  tools  to  a 
reasonable  depth 

-  Need  to  have  the  "right"  people  participating  in  the  WG  (people  actively  involved 
in  analysis  and  the  development/use  of  tools)  to  answer  these  questions 

•  SIMTECH97  WG  didn’t  pay  sufficient  attention  to  modeling  and  analysis 
needs  such  as  dealing  with  uncertainty,  W& A,  etc.  but  focused  on 
information  technology  such  as  software  and  distributed  system  issues 


The  WG  felt  that  SIMTECH97  didn’t  pay  sufficient  attention  to  predicting  the  future  environment 
and  suggests  that  the  next  workshop  may  want  to  invite  one  or  more  futurists  to  provide  input  for 
developing  a  consensual  view  of  the  future  environment. 

As  was  mentioned  on  the  previous  chart,  the  WG  felt  that  the  SIMTECH97  WG  did  not 
adequately  address  who  the  analysts  were,  their  needs  and  requirements  and  the  tools  that  could  satisfy 
their  needs  and  requirements.  The  WG  recommends  that  the  WG  at  the  next  workshop  be  composed 
primarily  of  active  practitioners  not  M&S  managers  in  order  to  answer  these  questions. 

Lastly,  the  WG  felt  that  the  SIMTECH97  WG  over  emphasized  technology  and  didn't  pay  enough 
attention  to  M&S  needs  such  as  addressing  uncertainty,  W&A  knowledge  base,  etc. 
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Other  Findings  and  Recommendations 


•  Address  whether  tool  developers  and  users  have  to  be  software  engineers 
to  build  simulations  and,  if  so,  then  how  do  they  maintain  connection  to 
the  OR/analysis  community  to  understand  and  meet  their  needs? 

•  Re-examine  the  future  roles  involved  in  simulation  development,  analysis 
and  use  (e.g.,  decision  maker,  staffer,  analyst,  simulation  developer, 
simulation  modifier). 


The  WG  was  concerned  with  whether  software  development  and  OR  tool  development/usage  have 
both  become  so  technology  intensive  that  it  may  not  be  possible  to  find  people  with  both  skills  to 
develop  M&S.  If  that  is  so,  then  attention  needs  to  be  paid  as  to  how  M&S  will  be  developed  so  they 
adequately  address  OR  concerns. 

A  related  issue,  is  for  the  next  workshop  to  re-examine  the  roles  involved  in  M&S  development 
and  in  its  use  for  analysis.  First,  if  M&S  development  needs  to  be  done  by  software  engineers  then  how 
well  will  they  be  responsive  to  OR  needs.  Secondly,  if  there  are  high  level  tools  that  make  it  easy  for 
analysts  and/or  decision  makers  and  staffers  to  modify  M&S  or  compose  a  new  M&S  from  reusable 
components,  what  kinds  of  help  do  they  need  in  understanding  tradeoffs,  performing  W&A,  etc. 
Include  consideration  of  decision  makers  having  to  make  time  critical  decisions  such  as  for  COA 
analysis.  Also  take  into  consideration  the  fact  that  in  the  future  there  might  not  be  such  a  clean  break 
between  M&S  for  analysis  (e.g.,  JWARS)  and  M&S  for  training  (e.g.,  JSIMS).  There  may  be  more  and 
more  of  a  need  to  analyze  training  exercises  and  perhaps  the  development  of  new  tools  to  support  such 
analyses. 
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Report  of  the 

Soft  Factors  Working  Group 


gjjgg 


The  Soft  Factors  Working  Group  opened  with  a  brief  retrospective  of  the  work  of 
SIMTECH  97  and  then  the  chair  laid  out  the  tasks  the  group  was  responsible  for  addressing  for 
the  afternoon  briefing  session.  The  group  was  charged  with  reviewing  the  analysis  and 
recommendations  of  the  Soft  Factors  Working  Group  at  SIMTECH  97.  The  group  was  to 
assess  what  its  SIMTECH  97  counterparts  did  right,  what  was  misidentified  and  what  was 
omitted  from  the  analysis.  Based  on  this  assessment,  the  group  was  to  draw  lessons  learned 
and  propose  a  series  of  recommendations  to  guide  future  efforts  in  soft  factors. 
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Participants  (1  of  2) 


Dick  Hayes,  Chair 
Lashon  Booker,  Co-Chair 

MAJ  Jack  Jackson 
John  Shepherd 
Doug  Kupersmith 
Clayton  Thomas,  FS 
Charlie  Leake 
Peter  Campbell 
Brian  Smith 


Following  the  background  review,  the  members  of  the  group  introduced  themselves  and  made  their 
initial  comments  on  the  SIMTECH  97  work.  The  participants  were  concerned  that  soft  factors  as  a  group 
were  not  well  defined,  that  the  enhancements  provided  by  soft  factors  were  not  tangible  and  thus  their 
utility  is  hard  to  defend  to  sponsors,  there  was  no  institutional  advocate  for  soft  factors  and  it  was 
commonly  assumed  that  soft  factors  were  the  purview  of  the  services.  Participants  were  also  concerned 
that  the  modeling  of  soft  factors  was  characterized  by  the  substitution  of  goals  for  standards  in  dealing 
with  seemingly  insolvable  problems.  Additional  points  included  the  lack  of  effort  to  record  human 
reactions  and  to  develop  data  to  support  soft  factors  and  the  hazards  of  using  incomplete  or  inappropriate 
soft  factors  or  soft  factor  data. 

The  group  moved  into  a  general  discussion,  based  in  part  by  their  initial  comments,  and  in  part  on 
the  comments  by  the  chair.  It  was  stated  that  the  lack  of  institutional  support  has  made  it  difficult  for 
objectives  and  standards  to  be  developed  because  they  are  not  necessarily  based  in  a  specific  requirement. 
Though  requirements  or  standards  for  soft  factors  could  be  drawn  from  existing  directives  in  support  of 
JV2010,  it  was  important  to  recognize  the  tradeoffs  between  descriptive  versus  prescriptive  modeling  and 
to  distinguish  broad  goals  from  specific  solutions. 

One  of  the  difficulties  in  soft  factors  is  the  lack  of  information  on  group  or  social  interactions  as 
opposed  to  individual  psychology.  If  command  and  staff  interactions  are  taken  as  social  processes,  then 
it  enters  the  realm  of  the  sociologists  and  anthropologists.  Such  disparities  point  to  the  fact  that  the 
science  behind  soft  factors  is  weak.  If  the  science  is  weak,  the  engineering  is  weak  and  consequently  the 
best  modeling  is  a  relatively  poor  tool.  As  the  sciences  supporting  soft  factor  elements  tend  to  produce 
probabilities  and  distributions  rather  than  single  point  data,  they  are  fraught  with  uncertainty.  The  result 
is  that  movement  within  the  distribution  tends  to  aggregate  uncertainty  within  and  between  models. 
Unfortunately,  the  cumulative  uncertainty  may  overwhelm  the  knowledge  produced  by  modeling. 

The  chair  reinforced  the  above  points  by  examining  the  differences  between  simple  and  complex 
decisions.  As  simple  decisions  tend  to  gain  parties,  they  slow  down  but  maintain  the  same  level  of 
quality.  However,  in  the  case  of  complex  decisions,  the  addition  of  parties  also  slows  the  process,  but 
does  improve  the  quality  of  the  decisions.  While  the  rule  has  an  important  consequence  for  modeling 
decision  making,  it  exists  at  a  level  of  generality  which  makes  it  difficult  to  adequately  represent. 
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Participants  (2  of  2) 

Dick  Hayes,  Chair 
Lashon  Booker,  Co-Chair 

MAJ  Jack  Jackson 
John  Shepherd 
Doug  Kupersmith 
Clayton  Thomas,  FS 
Charlie  Leake 
Peter  Campbell 
Brian  Smith 


Following  that  thread  in  the  discussion,  a  participant  pointed  out  that  part  of  the  resolution 
to  this  problem  was  to  decide  on  what  science  (sociology,  anthropology,  psychology,  etc.)  was 
to  be  used,  select  an  area  of  soft  factors  to  focus  on,  determine  the  kinds  of  experiments  that 
were  supported  by  the  relevant  methodologies,  and  then  choose  among  the  sub-disciplines  for 
further  analysis  and  data  generation  for  modeling. 

It  will  be  important  to  identify  problems,  which  are  currently  of  concern,  that  have  been 
allowed  to  lag.  The  identification  of  these  problems  can  help  to  focus  resources  and  begin  to 
develop  some  institutional  support  for  soft  factors. 

Experimentation  is  a  difficult  area  to  push  forward.  The  battle  labs  and  other  simulation 
infrastructure  do  very  little  true  experimentation.  The  tendency  is  to  provide  systems  to 
potential  users  and  then  observe  them  using  the  system.  The  lack  of  structured 
experimentation  is  hampering  the  generation  of  reliable  soft  factors  data.  The  number  of 
instrumented  ranges  is  declining  and  facilities  for  structured  data  collection  are  being  reduced. 

Two  participants  pointed  out  that  the  collection  of  data  is  fine,  but  that  which  has  been 
collected  is  difficult  to  access  and  the  existence  of  much  of  the  research  is  not  commonly 
known.  A  clearinghouse  of  soft  factor  data  and  studies,  plus  a  place  for  interaction  such  as  a 
chat  room  would  be  useful.  Another  pointed  out  that  the  Army  Modeling  and  Simulation 
Office  has  an  archive  of  studies  which  may  be  useful. 

It  was  pointed  out  that  many  similar  recommendations  have  been  made  in  the  past,  but 
have  not  been  acted  upon.  This  applied  to  the  range  of  discussion,  not  just  the  clearinghouse 
and  chat  room.  The  concern  was  that  without  some  institutional  support  or  sponsorship,  the 
recommendations  developed  by  the  group  would  not  receive  supporting  actions. 
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SIMTECH  97  Recommendations  (1  of  2) 

•  Establish  an  Operational  Performance  Measurement  Program  for  soft 
factors 

•  Report  on  what  is  now  being  done/not  done  and  prioritization  of  needs 

•  Adopt  an  experimental  paradigm 

-  Maximize  data  validity 

-  Apply  quasi-experimental  designs 

•  Create  a  data  registry 
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SIMTECH  97  Recommendations  (2  of  2) 

•  Build  on  existing  tools  for  evaluating  quality  of  command  and  control 

•  Require  transfer  plans  for  simulations  developed  for  multiple  users 

•  Focus  community  attention  on  soft  factors 

•  Broaden  behavioral  science  base:  expertise  and  education 

•  Emphasize  group  and  organizational  behavior 
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Assessment  of  SIMTECH  97 


•  What  did  it  do  well? 

-  Causal  factors,  modeling  issues  and  soft  effects  identified  correctly 

-  State  of  the  art  assessed  correctly 

•  What  did  it  not  do  well? 

-  Understated  the  influence  of  information  on  the  battlefield 

-  Incomplete  specification  of  the  solution  space 


Eggg 


The  group  determined  that  the  causal  factors,  modeling  issues  and  soft  effects  identified  at 
SIMTECH  97  were  still  appropriate  and  the  lists  were  complete.  The  discussion  focused 
around  the  ability  to  accurately  model  the  soft  effects  and  whether  the  definitions  of  the  soft 
effects  translated  to  traditional  modeling  terms,  notably  the  idea  of  disruption.  They  also  felt 
that  the  SIMTECH  97  work  continued  to  accurately  reflect  the  state  of  the  art.  There  was 
some  concern  over  the  slow  pace  of  advancement  in  the  areas  of  cognitive  processes  and 
human  factors,  especially  at  the  group  level. 

In  addressing  the  shortcomings  of  SIMTECH  97,  the  group  felt  that  the  influence  of 
information  on  the  battlefield  was  understated.  The  most  significant  shortcoming  was 
understating  the  impact  of  the  increase  in  speed,  both  in  moving  information  to  the  battlefield 
and  processing  in  general.  The  group  also  felt  that  the  solution  space  was  insufficiently 
specified,  thus  there  was  insufficient  structure  to  begin  a  systematic  inclusion  of  soft  factors. 
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Lessons  Learned 


•  Soft  factors  over  aggregated;  did  not  penetrate  the  problem. 

•  Failure  to  forecast  the  speed  of  improvement  in  information  technologies 

•  Failure  to  foresee  the  importance  of  information  warfare 

•  Failure  to  recognize  the  importance  of  context 

•  Failure  to  identify  institutions  to  support  and  to  carry  through 
recommendations 


The  group  felt  that  soft  factors  were  generally  over  aggregated,  making  it  difficult  to  drill 
down  to  the  level  of  detail  that  would  have  a  noticeable  impact  on  modeling.  Aggregation  also 
makes  it  difficult  to  show  sponsors  and  supporting  institutions  the  value  added  from  modeling 
soft  factors.  The  group  reinforced  its  belief  that  the  failure  of  SIMTECH  97  to  address 
improvements  in  information  technologies  was  also  a  lesson  learned,  and  went  on  to  include 
the  failure  to  foresee  the  impact  of  issues  loosely  defined  as  information  warfare.  Though 
deception  and  false  images  could  be  simulated  and  included  in  models,  more  attention  needed 
to  be  paid  to  the  defeat  of  sensors  and  reliance  on  automated  decisions. 

The  group  felt  that  importance  needed  to  be  placed  on  putting  soft  factors  into  context. 
The  value  of  soft  factors  is  enhanced  by  putting  them  into  a  useful  context.  Context  would 
also  aid  the  structuring  of  data  collection  and  experimentation. 

Finally,  the  group  recognized  the  importance  of  identifying  institutions  to  support  and 
carry  through  with  the  recommendations.  The  discussion  focused  on  the  apparent  lack  of 
specific  requirements  for  soft  factors  as  preventing  institutions  from  stepping  forward,  even 
those  who  have  charter  responsibility.  The  soft  factor  problem  may  be  so  difficult  that 
developing  or  identifying  sponsors  may  be  a  continuing  problem. 
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Current  Recommendations  (1  of  2) 

•  Need  to  anchor  to  a  clear  articulation  of  requirements;  motivate 
ownership  and  sponsorship 

•  Need  to  prioritize  the  soft  factors  and  how  they  effect  the  solution  space(s) 

•  Soft  factors  may  require  different  approaches  and  techniques 

-  Identify  useful  knowledge  and  build  on  it  (cognitive  science,  stochastic 
representation,  etc.) 

-  Incorporate  learning  algorithms 

-  Draw  from  dynamic  game  theory 


Soft  factors  need  to  be  anchored  to  clearly  articulated  requirements  so  as  to  develop  a 
sense  of  institutional  ownership.  The  current  substitution  of  goals  for  standards  is  hindering 
that  process. 

Though  SEMTECH  97  made  a  complete  list  of  factors,  it  is  important  to  move  forward 
with  a  prioritization  of  those  factors.  The  impact  of  soft  factors  on  the  definition  of  a  solution 
space  needs  to  be  determined,  which  will  further  enhance  their  utility  and  the  development  of 
institutional  ownership. 

The  individual  soft  factors  will  require  a  separate  approach  in  terms  of  identifying 
available  knowledge  and  developing  strategies  and  efforts  to  build  on  that  knowledge  base. 
Research  and  development  of  learning  algorithms  will  also  require  separate  approaches  so  that 
the  algorithms  can  better  address  the  subtleties  of  each  factor,  especially  in  the  case  of 
aggregated  factors.  Separate  approaches  and  techniques  may  also  be  useful  in  the  case  of 
dynamic  game  theory. 
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Current  Recommendations  (2  of  2) 


•  Use  an  experimental  paradigm 

-  Control  most  factors 

-  Manipulate  selected  issues 

-  Accumulate  data  and  parameters 

•  Create  an  ongoing  working  group 

-  Build  a  clearinghouse  on  the  Web  and  provide  contractor  support; 

-  include  a  chat  room,  interacting  with  DMSO  and  other  relevant  organizations 


The  group  recommended  the  use  of  an  experimental  paradigm  to  provide  control  for  most 
factors,  to  manipulate  selected  issues  so  as  to  determine  their  impact  on  causal  relationships 
and  to  aid  in  the  accumulation  of  data  and  the  development  of  parameters.  The  experimental 
paradigm  would  begin  to  lend  structure  to  the  process  of  including  soft  factors  and  is  an 
important  primary  step. 

Finally,  the  group  decided  that  an  ongoing  working  group  should  be  created  and  that  in 
support  of  that  working  group,  a  clearinghouse  of  data  and  studies  should  be  established  on  the 
web.  The  clearinghouse  should  be  supported  by  contractors  to  maintain  currency  and 
usability.  As  an  adjunct,  the  clearinghouse  would  also  have  a  chat  room  to  aid  the  interaction 
among  modelers,  DMSO  and  other  relevant  organizations. 
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Soft  Factors,  Effects  and  Modeling 


•Top  level  soft  factors 

-  Cognitive  processes 

-  Individual 

-  Group 

-  Human  factors 

-  Individual 

-  Group 

-  Organizational  factors 

-  Combat  induced 
(environmental)  factors 

•Related  modeling  issues 
-Integration 
-Aggregation 
-Representation 


Simulation 

Model 


x 


-H 


•Soft  effects 

-  Delay 

-  Disruption 

-  Cohesion 

-  Reconstitution 

-  Synchronization 

-  Surprise 

-  Effectiveness 


The  last  slide  depicts  the  basic  relationship  between  soft  factors,  related  modeling  issues 
and  soft  effects  being  fed  through  a  simulation  model.  The  slide  shows  that  aggregated  soft 
factors,  when  filtered  through  a  simulation  model  and  combined  with  other  basic  modeling 
issues,  produces  a  series  of  soft  effects  which  can  be  incorporated  into  other  levels  of 
modeling.  The  slide  lists  the  soft  factors  identified  in  SEMTECH  97,  which  the  SIMTECH  07 
participants  endorsed  as  being  correct  and  complete. 
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Phil  Dickinson,  Co-Chair 
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Carl  Russell 
Cy  Staniec 
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SIMTECH  ‘97  Revisited  -  DWG 


“Without  Data  we  are  nothing”  La  Berge  ‘87 
SIMTECH  97  Data  Group  developed  two  viewpoints 

-  The  ability  to  capture  quality  data  and  make  it  available  to  modelers  was  severely 
limited 

-  The  environment  to  generate  good  representative  data  for  many 
weapons/environments/scenarios  is  severely  limited 

A  top  level  review  today  seems  to  indicate  that: 

-  The  DMSO-led  effort  is  cataloging  authoritative  data  sources 

-  There  is  no  ongoing  effort  to  build  a  DoD-wide  integrated  database 

-  In  recent  years  major  OTs  are  less  prevalent/less  instrumented  and  therefore  less 
hard  data  is  available  —  and  the  world  has  changed 

-  No  effective  transducer  for  data  from  targets  of  opportunity  (AWEs  and  exercises) 
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DWG  -  Type  I  Errors,  Pd 


Data...will  not  change  radically  in  89  -  97  window 

-  Clearly  the  “outbreak  of  peace”  demands  different  sorts  of  data  and  modeling. 

-  What  are  the  current  needs? 

Overt  structure  that  links  all  members  of  data/modeling  team 

-  No  such  DoD-wide  structure  exists 

(but  implemented  within  major  programs) 

-  Failed  to  recognize  the  internet  as  the  logical  vehicle  to  achieve  a  networked 
community 
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DWG  -  Type  II  Errors,  Pfa 


Characterize  EM  environment 

-  Information  Warfare  is  the  current  buzz  word,  but  little/no  progress  toward 
characterizing  the  EM  environment 

-  EM  was  too  restrictive  to  characterize  environmental  effects 

Missed  Simulation  Based  Acquisition  (SBA) 

-  Changes  the  fundamental  relationship  between  M&S  and  field  data  collection  — 
instead  of  providing  data  to  load  models,  field  tests  will  be  used  more  to  validate 
models 
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Assessment  of  SIMTECH  ‘97  Data  WG 


Complex  models...Demand  disciplined  process  of  definition,  design, 
implementation  and  documentation  —  which  is  not  compatible  with  an  ad 
hoc  approach 

-  They  demand  rigorous  systems  engineering 

Moving  from  monolithic  models  to  distributed  federated  models  demands 
increased  system  engineering 

-  Selected  flag  ship  models  are  working  the  problem  well  —  but  this  is  not  the 
common  approach 
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Recommendations 


Data  important 

-  Data  sources  drying  up 

•  Few  dedicated  experiments  to  generate  data 

•  Fewer  instrumented  OTs 

•  Data  outsourced/vendor  generated  (ownership) 

-  Incomplete  process  to  acquire  and  manage  available  data 

•  Inadequate  data  requirements  process 

•  No  disciplined  processes  to  harvest  useful  data  from  AWEs  and  exercises 

This  is  a  problem  that  needs  to  be  addressed  by  the  M&S  Working  Group 
(chaired  by  DMSO) 
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Other  Findings  and  Recommendations  (1  of  2) 


•  Need  Godfather  to  make  institutional  things  happen 

-  Need  to  create  integrated  data  process  across  DoD  to  support  SBA 

-  Energize  the  EXSIM  to  focus  the  OSD  leadership  on  the  need  for  robust  data 

•  Modify  TOR  to  include  the  individuals  and  organizations  who  generate 
the  data  in  defining  the  solution  (AWEs,  FTXs  etc.) 
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Other  Findings  and  Recommendations  (2  of  2) 

•  Where  are  the  “experimenters”  who  generate  the  data  to  validate  models? 

-  Look  for  innovative  techniques  to  replace  dedicated  experiments 

•  Would  be  useful  to  include  the  decision  makers  in  the  process  —  the 
people  who  spend  money  based  on  M&S  results  —  not  just  the 
proponents  who  “like  M&S” 


Egg 
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Lessons  Learned  from  SIMTECH  ‘97 


•  S97  DWG  focus  was  too  narrow 

-  Composed  mostly  of  data-generating  people 

-  Need  producers  and  consumers 

-  Don’t  focus  WG  composition  too  narrowly 

•  Complexity  of  the  data  requirements  has  increased 

-  Additional  data  sources 

-  Multiple  mission  requirements 

-  International  data  requirements 

-  State  Department  in  charge  of  peace-keeping  team 

•  The  data  process  posited  in  S97  has  not  materialized 

-  Bigger  problem  today 
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Lessons  Learned 


•  Watershed  events  occurred  since  S97 

-  Warfighting 

•  Wall  came  down 

•  Peacemaking,  Operations  Other  Than  War 

•  Desert  Storm 

•  NBC 

-  Technology 

•  Distributed  processing 

•  Computer  hardware 

•  Internet 

•  Even  with  this  turbulence 

-  change  has  taken  place  majestically 
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Terms  of  Reference 

Military  Operations  Research  Society  (MORS) 
Simulation  Technology  2007  (SIMTECH  2007) 


Background 

In  1987,  MORS  was  approached  by  representatives  of  the  Army,  Air  Force,  and 
Joint  Staff  to  assess  the  appropriate  direction  for  the  sponsors  to  take  in  military 
simulation  technology,  over  the  following  decade,  in  support  of  the  analysis  community. 
First,  the  participants  were  to  identify  and  prioritize  deficiencies  in  military  simulation. 
Based  on  an  understanding  of  these  deficiencies,  appropriate  technologies  were  to  be 
identified  and  assessed  that  have  the  potential  to  ameliorate  these  deficiencies  in  the  near- 
and  mid-term  (out  to  1997).  Finally,  a  set  of  recommended  follow-on  actions  were  to  be 
proposed  to  implement  the  workshop’s  findings  and  recommendation. 

In  order  to  satisfy  those  objectives,  three  workshops  were  held.  These  culminated 
with  a  set  of  recommended  actions  to  ameliorate  shortfalls  in  four  key  areas:  life  cycle 
management,  analyst  workbench,  “soft”  factors,  and  data. 

Current,  related  DoD  programs  to  address  simulation  deficiencies  are  underway  in 
the  areas  of  analysis  (e.g.,  the  Joint  Analysis  Model  Improvement  Program  (JAMIP)), 
training  (e.g.,  Joint  Simulation  System  (JSIMS)),  and  acquisition  (e.g.,  Simulation  Based 
Acquisition). 

Goals  and  Objectives 

The  overarching  goal  of  SIMTECH  2007  is  to  promote  more  effective  dialogue 
between  the  users  of  modeling  and  simulation  (M&S)  and  the  M&S  technology 
community.  Consistent  with  that  goal,  the  objectives  of  SIMTECH  2007  are  to: 

(1)  Review  and  assess  the  findings  and  recommendations  that  were  made  by  the 
participants  at  SIMTECH  97.  This  will  include  an  assessment  of  the  accuracy  with  which 
the  participants  were  able  to  identify  and  characterize  deficiencies  and  major  M&S  trends 
and  issues  over  the  past  decade,  the  appropriateness  of  the  recommendations  that  the 
participants  made,  and  the  extent  to  which  the  recommendations  were  implemented. 

(2)  After  consideration  of  the  lessons  learned  from  the  retrospective  assessment, 
the  participants  will  identify  and  prioritize  the  needs  of  the  users  of  military  M&S  and 
assess  the  probable  evolution  of  M&S  technology  over  the  next  decade.  Based  on  an 
understanding  of  these  prioritized  needs  and  technology  trends,  appropriate  technologies 
will  be  identified  and  assessed  that  have  the  potential  to  ameliorate  identified  deficiencies 
looking  out  to  the  year  2007.  The  participants  will  identify  a  set  of  recommendations  to 
address  these  perceived  deficiencies. 
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Scope 


SIMTECH  2007  will  consider  M&S  in  support  of  analysis,  training,  and 
acquisition.  This  represents  an  expansion  of  the  scope  of  SIMTECH  97  which  was 
restricted  to  the  support  of  analysis.  Technologies  will  be  considered  in  the  near-term 
(e.g.,  2002  timeframe)  and  longer-term  (i.e.,  out  to  2007).  The  activity  also  may  consider 
issues  that  relate  to  technology  transfer  into  organizations  (e.g.,  organizational  mission 
and  structure,  training,  facilities). 

Activities 


The  objectives  of  SIMTECH  2007  will  be  achieved  by  convening  two  workshops. 

•  Workshop  I 

Workshop  I  will  be  held  in  Northern  Virginia  in  December  1997.  As  a  context  for 
the  workshop,  a  read-ahead  package  will  be  distributed  that  includes,  inter  alia,  the  final 
products  that  were  developed  in  SIMTECH  97.  Extensive  use  will  be  made  of  the  Internet 
to  identify  and  make  accessible  relevant  information. 

The  agenda  for  the  three  day  unclassified  workshop  will  be  as  follows.  The 
workshop  will  begin  on  the  morning  of  Day  1  with  a  plenary  session  that  includes  a 
presentation  on  the  purpose  and  structure  of  the  workshop,  a  keynote  address  by  Ed 
Brady  that  reviews  the  key  findings  and  recommendations  that  emerged  from  SIMTECH 
97,  and  a  presentation  by  an  expert  in  modeling  and  simulation  (M&S)  that  reviews  the 
key  events  in  the  area  of  M&S  that  have  occurred  between  the  years  1987  and  1997.  The 
participants  will  then  breakout  into  the  four  groups  that  were  employed  in  SIMTECH  97: 
life  cycle  management,  analyst  workbench,  “soft”  factors,  and  data.  To  the  extent 
feasible,  these  breakout  groups  will  be  co-chaired  by  the  individuals  who  chaired  the 
original  groups  in  SIMTECH  97.  These  groups  will  conduct  deliberations  to  address  the 
first  set  of  objectives  identified  above.  In  addition,  a  Synthesis  Group  will  be  established 
to  identify  and  characterize  observations  that  cut  across  the  deliberations  of  all  the 
breakout  groups.  In  the  morning  of  Day  2,  these  groups  will  report  out  on  the  results  of 
their  retrospective  assessments. 

Immediately  after  the  retrospective  working  group  brief  outs  on  Day  2,  a  panel  of 
experts  will  address  the  latest  initiatives,  objectives,  and  priorities  of  using  M&S  in 
support  of  analysis,  training,  and  acquisition.  These  experts  will  be  the  chairs  of  the 
EXCIMS  councils:  Jim  Johnson  and  Vince  Roske,  Analysis;  Pat  Sanders,  Acquisition, 
Lou  Finch  and  the  Director,  J7,  Joint  Staff,  Training.  This  panel  will  provide  a  context  for 
the  subsequent  breakout  groups  by  providing  a  vision  of  M&S  technology  and 
requirements  out  to  2007. 

After  the  panel,  the  participants  will  breakout  into  two  clusters.  Cluster  A  will 
identify  and  prioritize  the  functional  needs  for  M&S  in  the  functional  areas  of  analysis, 
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training,  and  acquisition1,  using  the  information  and  guidance  given  to  them  by  the  panel 
as  a  basis  for  addressing  the  needs  beyond  the  currently  projected  initiatives.  Cluster  B 
will  perform  a  technology  assessment  of  the  likely  evolution  of  M&S  technology 
(civilian  and  DoD)  over  the  next  decade.  At  the  discretion  of  the  Technology  Cluster 
leadership,  the  cluster  may  be  divided  into  several  subgroups  (e.g.,  information 
technology;  representation  of  environment;  representation  of  human  behavior;  “new 
sciences”). 

The  leaders  of  the  breakout  groups  will  brief  out  their  results  on  the  afternoon  of 
Day  3.  The  workshop  will  conclude  with  a  presentation  by  the  Synthesis  Group  that 
provides  a  synoptic  overview  of  the  proceedings.  The  leaders  of  the  breakout  groups  will 
remain  until  noon  of  the  following  day  to  annotate  the  vugraphs  that  they  used  to  brief 
out  their  results. 

•  Workshop  II 

Workshop  II  will  be  held  in  the  Spring  of  1998.  As  context  for  Workshop  n,  a 
read  ahead  package  will  be  assembled  and  distributed  prior  to  the  workshop  that  includes, 
inter  alia,  the  annotated  briefings  that  were  developed  in  Workshop  I.  Again,  the  Internet 
will  be  used  as  the  primary  mechanism  for  disseminating  information. 

The  Agenda  for  the  three  day  workshop  will  be  as  follows.  The  workshop  will 
begin  on  the  morning  of  Day  1  with  a  Plenary  session  that  includes  a  presentation  on  the 
purpose  and  structure  of  Workshop  II  and  a  summary  of  the  major  results  from  Workshop 
I.  The  participants  will  then  breakout  into  the  functional  work  groups  employed  in 
Workshop  I  with  the  original  groups  augmented  with  appropriate  experts  from  the 
technology  cluster.  During  the  course  of  Workshop  n,  the  augmented  functional  breakout 
groups  will: 

-  Review  and  validate  the  functional  needs  identified  and  prioritized  in  Workshop 

I; 

-  Translate  the  prioritized  functional  needs  into  technical  needs; 

-  Identify  potential  military  simulation  technology  shortfalls  by  comparing  the 

technical  needs  to  the  results  of  the  M&S  trends  analyses  from  Workshop  I; 

-  Prioritize  these  shortfalls  and  recommend  actions  to  ameliorate  them. 

On  the  afternoon  of  Day  3,  the  leaders  of  the  breakout  groups  will  brief  out  their 
major  findings  and  recommendations.  The  workshop  will  conclude  with  a  presentation  by 
the  Synthesis  Group  which  will  identify  and  discuss  similarities  and  differences  among 
the  findings  and  recommendations  of  the  breakout  groups.  The  breakout  group  leadership 
will  remain  until  noon  of  the  following  day  to  complete  the  annotation  of  their  vugraphs. 


1  During  the  course  of  the  Workshop,  it  may  prove  desirable  to  divide  the  analysis  group  into  two 
subgroups:  analysis  in  support  of  operations  (e.g.,  course  of  action  analysis  and  mission  rehearsal  and 
assessment)  and  analysis  in  support  of  high  level  decision  making  (e.g.,  force  structure  and  force  mix 
analyses;  support  to  planning,  programming,  and  budgeting  activities). 
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Products 


The  primary  product  of  the  workshops  will  be  an  annotated  briefing  booklet  that 
includes  an  overview  of  the  workshops  and  individual  chapters  documenting  the 
deliberations  of  the  working  groups.  In  addition,  an  overview  white  paper  will  be 
prepared  that  is  suitable  for  publishing  in  Phalanx  and  a  briefing  will  be  prepared  for  the 
MORS  Sponsors  and  the  66th  MORSS. 

Membership 

The  workshop  will  be  organized  by  a  committee  that  consists  of  Stuart  Starr 
(chair).  Bob  Orlov  (co-chair),  Howard  Carpenter,  Denis  Clements,  Bob  Statz,  and  Ken 
Konwin  (MORS  Workshop  Advisor).  The  chairs  will  control  the  attendance  so  that  it  is 
approximately  120  per  workshop.  Attendees  will  be  sought  from  both  the  SIMTECH  97 
workshops,  for  continuity,  and  new  participants,  to  introduce  fresh  perspectives.  These 
new  participants  will  be  drawn  from  council  functional  working  group  members, 
appropriate  MORS  working  groups,  and  experts  from  advanced  M&S  fields  outside  of 
the  usual  DoD  operations  research  and  related  disciplines. 

Chairs  for  the  first  breakout  session  of  Workshop  I  will  include:  Bruce  Bennett 
and  Bob  Statz  (Life  Cycle  Management);  Iris  Kameny  and  John  Gilmer  (Analyst 
Workbench);  Dick  Hayes  and  Lashon  Booker  (“Soft”  Factors);  and  Phil  Dickinson  and 
Carl  Russell  (Data).  Chairs  for  the  second  breakout  session  of  Workshop  I  remain  to  be 
determined  but  candidates  include  Cy  Staniec/Hank  Dubin/Russ  Richards  (Analysis); 
LTC  Coble/Zach  Furness  (Training);  and  Dick  Helmuth/Gary  Jones  (Acquisition). 
Candidates  for  leadership  of  the  Technology  Assessment  Cluster  include  Denis  Clements, 
and  Dell  Lunceford.  The  Synthesis  Group  will  include  Paul  Davis,  Clayton  Thomas,  and 
Vem  Bettencourt. 

In  Workshop  II  the  chairs  for  the  functional  breakout  groups  will  be  retained  from 
the  comparable  breakout  groups  from  Workshop  I,  to  the  extent  feasible. 

Schedule  and  Fees 


Workshop  I  will  be  held  in  McLean,  VA,  at  GRCI  during  the  third  week  of 
December  (16  - 19  December).  Workshop  II  will  be  held  in  McLean,  VA,  at  Booz,  Allen 
&  Hamilton  (BAH)  during  the  first  week  of  March  1998  (4-6  March).  The  fee  is 
anticipated  to  be  $175  for  federal  government  employees  and  $350  for  all  others,  for  each 
workshop,  with  a  $50  reduction  if  the  participant  attends  both  workshops.  This  fee 
structure  will  be  part  of  the  management  plan  developed  by  the  MORS  Office,  the  MORS 
Workshop  Advisor,  and  the  Workshop  Co-chairs. 
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A  selected  set  of  major  milestones  is  as  follows: 


1  Oct 

15  Oct 
20  Oct 
3  Nov 

19  Dec 

8  Jan 

20  Jan 


2  Feb 
6  March 


6  April 
6  May 


June 

6  August 

7  Sept 


Complete  identification,  selection,  and  gaining  acceptance  of  co-chairs  for 

breakout  groups  of  Workshops  I  and  II 

Send  out  first  round  of  invitations  to  Workshop  I 

Develop  and  disseminate  product  outline  for  Workshop  I 

Begin  to  populate  the  read  ahead  Internet  page  for  the  workshop 

Generate  rough  draft  of  Executive  Summary;  generate  preliminary 

annotated  vugraphs  for  each  working  group 

Send  out  invitations  to  Workshop  II 

Submit  draft  of  Executive  Summary  and  annotated  vugraphs  for 
Workshop  I  to  the  MORS  Office;  if  deemed  appropriate,  generate  draft 
article  to  PHALANX  editor 

Disseminate  draft  materials  from  Workshop  I  and  other  read  ahead 
material  to  participants  in  Workshop  II 

Generate  rough  draft  of  complete  Executive  Summary  (encompassing 
Workshops  I  and  II),  along  with  preliminary  annotated  vugraphs  for  the 
working  groups  in  Workshop  II 

Submit  article  to  Phalanx  editor;  submit  draft  of  total  Executive  Summary 

and  annotated  vugraphs  for  both  Workshops  to  MORS  Office 

Receive  comments  on  draft  from  key  personnel;  brief  Sponsors; 

incorporate  Sponsor  feedback  in  executive  summary;  post  approved 

Executive  Summary  on  web  site 

Brief  out  the  results  of  SIMTECH  2007  at  66th  MORSS 

Submit  smooth  report  to  printer 

Publish  and  distribute  report 
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SIMTECH  2007 


Workshop  Organizers 


Position 

Name 

Affiliation 

Chair 

Stuart  Starr,  FS 

MURE 

Co-chairs 

Howard  Carpenter 

MURE 

Denis  Clements 

GRCI 

Lana  McGlynn 

DAMO,  USA 

Bob  Orlov 

J-8,  JS 

Bob  Statz 

BAH 

Col  "Crash"  Konwin 

DMSO 

Working  Group  Leadership 


Retrospective 

Life  Cycle  Management 

Bmce  Bennett,  RAND 

Bob  Statz,  BAH 

ml 

Analyst  Workbench 

Iris  Kameny,  RAND 

John  Gilmer,  Wilkes  University 

"Soft"  Factors 

Dick  Hayes,  EBR 

Lashon  Booker,  MITRE 

Data 

Phil  Dickinson,  Consultant 

Carl  Russell,  JNTF 

Looking  Forward 

Analysis 

Cy  Staniec,  Logicon 

Hank  Dubin,  OPTEC 

Russ  Richards,  MITRE 

Mark  Youngren,  MITRE 

Acquisition 

Gary  Jones,  DARPA 

Dick  Helmuth,  S  AIC 

Annie  Patenaude,  SAIC 

Dell  Lunceford,  DARPA 

Education  and  Training 

Phil  Abold,  A&B  Technology 

Zach  Furness,  MITRE 

Julia  Loughran,  Thoughtlink 

Technology  Assessments 

Dell  Lunceford,  DARPA 

Denis  Clements,  GRCI 

Julia  Loughran,  Thoughtlink 

Synthesis 

Paul  Davis,  RAND 

Clayton  Thomas,  FS,  USAF 
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Participants  in  SIMTECH  2007  Workshop  I 


Phil  Abold 
Jesse  Aronson 
Michael  Bailey 
Bruce  Bennett 
Lashon  Booker 
Peter  Campbell 
Howard  Carpenter 
Denis  Clements 
Gerald  E.  Cooper 
CAPT  Robert  Danberg 
CAPT  Lee  Dick 
Phil  Dickinson 
Norman  Eagleston 
Gary  Engel 
Dean  Free 
Robin  Frost 
Charles  Z.  Furness 
John  Gilmer 
Priscilla  Glasow 
Dick  Hayes 
Richard  Helmuth 
John  Hughes 
Susan  Iwanski 
MAJ  Jack  Jackson 
Gary  Jones 
Iris  Kameny 
Gary  Klein 
COL  Crash  Konwin 
Doug  Kupersmith 
Charlie  Leake 
Dell  Lunceford 
Joe  Manzo 
Richard  Maruyama 
LTC  Mike  McGinnis 
Lana  McGlynn 
Richard  Modjeski 
Richard  Nance 
Steve  Olson 
Robert  Orlov 
Ernie  Page 
Mel  Parish 
Annie  Patenaude 
F.  Russ  Richards 
Carl  Russell 


AB  Technologies 

SAIC 

MCCDC 

RAND 

MITRE 

Argonne  National  Lab 

MITRE 

GRCI 

UFSACAA 

COMTRAPAC 

OPNAV 

Consultant 

Sun  Microsystems 

Boeing 

Navy 

OSD 

MITRE 

Wilkes  University 
MITRE 

Evidence  Based  Research,  Inc. 

SAIC 

MITRE 

Northrup  Grumman 

TRAC  -  Monterey 

ARPA 

RAND 

MITRE 

DMSO 

S3I 

JWARS 

DARPA 

MITRE 

AMSO 

TRAC  Monterey 

AMSO 

Consultant 

Virginia  Tech 

Raytheon 

JS,J8 

The  MITRE  Corporation 

TRACWSMR 

DUSA 

MURE 

JNTF/SE 
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Bob  Sheldon 
John  Sheperd 
Brian  Smith 
Bob  Smith 
Marcy  Stahl 
Cy  Staniec 
Bob  Statz 
Scott  Swinsick 
Clay  Thomas 
Dave  Thomen 


S31 

USACAA 

Evidence  Based  Research,  Inc. 

Raytheon 

ThoughtLink,  Inc. 

Logicon 

BAH 

The  Boeing  Company 
HQUSAF  AFSAA 
SAIC 
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Participants  in  SIMTECH  2007  Workshop  II 


Phillip  L.  Abold 

AB  Technologies,  Inc. 

Thomas  L.  Allen 

IDA 

Dennis  R.  Baer 

Logicon 

Steven  C.  Bankes 

RAND 

Natalie  S.  Addison 

MORS 

Jesse  Aronson 

SAIC 

Thomas  J.  Baker 

Logicon 

Darrell  Bench 

US  Army  Test  &  Evaluation  Command 

Vernon  M.  Bettencourt  Jr.  FS 

Army  Model  and  Simulation  Office 

Mary  T.  Bonnet 

AFSAA/SAJ 

Jim  Brady 

S3I 

Gregory  A.  Brouillette 

United  States  Military  Academy 

James  N  Bexfield  FS 

Institute  for  Defense  Analyses 

Francis  A.  Bowers  HI 

The  MITRE  Corporation 

Fritz  H.  Brinck 

Naval  Surface  Warfare  Center 

Howard  J.  Carpenter 

The  MITRE  Corporation 

Denis  T.  Clements 

OSD  (PA&E)  GRCI 

Phil  Collins 

GRCI 

Paul  K.  Davis 

RAND 

Lawrence  L.  Dick 

OPNAV 

Gary  Q.  Coe 

Institute  for  Defense  Analyses 

Keith  P.  Curtis 

The  MITRE  Corporation 

Patrick  J.  Delaney 

Army  Model  and  Simulation  Office 

Kevin  Michael  Eveker 

Institute  for  Defense  Analyses 

Edward  A.  Feustel 

Institute  for  Defense  Analyses 

Charles  Zachary  Furness 

The  MITRE  Corporation 

Paul  F.  Goree 

Institute  for  Defense  Analyses 

Jeffrey  H.  Grotte 

Institute  for  Defense  Analyses 

W.  Dean  Free 

Chief  of  Naval  Operations  (N816) 

Dewey  P.  George) 

Soza  &  Company,  Ltd. 

David  Gross 

The  Boeing  Company 

Donald  F.  Haskell. 

US  Army  Evaluation  Analysis  Center 

William  O.  Hedgepeth 

GRC  International 

John  G.  Hughes 

The  MITRE  Corporation 

Kenneth  C.  Konwin 

DMSO 

Cynthia  Kee  LaFreniere 

Military  Operations  Research  Society 

Don  P.  Hodge 

Army  Materiel  Systems  Analysis  Activity 

Harold  L.  Jones 

Litton  TASC 

Jerry  A.  Kotchka 

Boeing 

Charles  R.  Leake 

USACAA 

Julia  Loughran 

ThoughtLink,  Inc. 

Lana  E.  McGlynn 

ODUSA  (OR) 

Peter  B.Merkle 

OSD  CP/CBD 

Richard  B.  Modjeski 

Consultant 

W.  H.  Lunceford 

DARPA 

Grcgoty  A.  McIntyre 

AFTT/ENS 

Robert  J.  Meyer 

NAWC-WPNS 
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Richard  E.  Nance 

VPI  &  SU 

John  R  Nelson 

Institute  for  Defense  Analyses 

Lawrence  F.  Olsen 

CINCLANTFLT  N7 

Christine  M.  Parnell 

Military  Operations  Research  Society 

Royce  H.  Reiss 

AFSAA/SAA 

David  Noble 

Evidence  Based  Research 

Robert  D.  Orlov 

The  Joint  Staff/ (J-8)WAD 

Anne  M.  Patenaude 

SAIC 

Roy  E.  Rice 

Teledyne  Brown  Engineering 

Patrick  I.  Riley 

US  Transportation  Command  J5-SC 

Corrina  Ross 

Military  Operations  Research  Society 

Patricia  A.  Sanders 

OUSD(A&T)/DTSE-E 

Albert  A.  Sciarretta 

Quantum  Research  International 

Vincent  P.  Roske  Jr. 

The  Joint  Staff  (J-8) 

Mr.  Thomas  W.  Ruth 

AMSAA 

Jeffrey  E.  Schofield 

Institute  for  Defense  Analyses 

Robert  S.  Sheldon 

S3I 

Brian  D.  Smith 

Evidence  Based  Research  Inc. 

Cyrus  J.  Staniec 

Logicon 

John  R.  Suatz  Jr 

Booz  Allen  &  Hamilton 

Clayton  J.  Thomas  FS 

AFSAA/SAN 

Patricia  Ann  Tracey 

Chief  of  Naval  Education  Training 

Marchelle  M.  Stahl 

ThoughtLink  Inc. 

Smart  H.  Starr 

The  MITRE  Corporation 

Glenn  H.  Tarbox 

Object  Sciences  Corp. 

David  S.  Thomen 

ETAS/S  AIC 

Stephen  C.  Upton 

Los  Alamos  National  Lab 

Richard  Weatherly 

The  MITRE  Corporation 

Larry  D  Welch  (Ret) 

Institute  for  Defense  Analyses 

Richard  I.  Wiles 

Military  Operations  Research  Society 

Mark  A.  Youngren 

The  MITRE  Corporation 

Robert  Eric  Weissler 

RAND 

Gary  A.  Yerace 

DMSO 
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SIMTECH  2007  Acronyms 


ACTD 

ACOM 

ADL 

ADS 

AI 

AMSO 

API 

ARPANET 

ATD 

ATM 

AWE 

BAH 

C2 

C3I 

C4I 

C4ISR 

CAD 

CAM 

CGF 

CINC 

CMMS 

COA 

COE 

CONUS 

CORBA 

COTS 

CPU 

CSL 

DARPA 

DDR&E 

DEVS 

DIF 

DH 

DIS 

DISA 

DL 


Advanced  Concept  Technology  Demonstrations 

US  Atlantic  Command 

Advanced  Distributed  Learning 

Advanced  Distributed  Simulation 

Artificial  Intelligence 

Army  Modeling  and  Simulation  Office 

Application  Programming  Interface 

[Defense]  Advanced  Research  Projects  Agency  Network 

Advanced  Technology  Demonstration 

Asynchronous  Transfer  Mode 

Army  Warfighting  Experiments 

Booz- Allen  Hamilton 

Command  &  Control 

Command,  Control,  Communications  and  Intelligence 

Command,  Control,  Communications,  Computers  and  Intelligence 

Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance 

and  Reconnaissance 

Computer  Aided  Design 

Computer  Aided  Manufacturing 

Computer  Generated  Force 

Commander-in-Chief 

Conceptual  Models  of  Mission  Space 

Course  of  Action 

Common  Operating  Environment 

Continental  United  States 

Common  Object  Request  broker  Architecture 

Commercial  Off  the  Shelf 

Central  Processing  Unit 

Condition  Specific  Languages 

Defense  Advanced  Research  Projects  Agency 

Director,  Defense  Research  and  Engineering 

Simulation  language  from  Univ.  of  Arizona 

Data  Interchange  Format 

Defense  Information  Infrastructure 

Distributed  Interactive  Simulation 

Defense  Information  Systems  Agency 

Distance  Learning 
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DMSO 

DoD 

DoE 

DSB 

DTSE&E 

DUSD 

DWG 

E&T 

EA 

EBR 

EM 

EXCIMS 

Flames 

FOM 

FTX 

GOTS 

GPS 

GRCI 

HBR 

HCI 

HLA 

IDA 

I/O 

DPT 

IT 

IW 

JMASS 

JROC 

JSIMS 

JTA 

JWARS 

M&S 

MIME 

MOE 

MOP 

MORS 

MRM 

MSRR 

NBC 


Defense  Modeling  and  Simulation  Office 
Department  of  Defense 
Department  of  Energy 
Defense  Science  Board 

Director,  Test,  systems  Engineering,  and  Evaluation 

Deputy  Under  Secretary  of  Defense 

Data  Working  Group 

Education  and  Training 

Executive  Agents 

Evidence  Based  Research 

Electromagnetic 

Executive  Council  on  Modeling  and  Simulation 

A  Simulation  Development  Environment 

Federation  Object  Model 

Field  Training  Exercise 

Government  Off  the  Shelf 

Global  Positioning  System 

GRCI  International,  Inc. 

Human  Behavior  Representation 

Human-Computer  Interface 

High  Level  Architecture 

Institute  for  Defense  Analyses 

Input/Output 

Integrated  Product  Team 

Information  Technology 

Information  Warfare 

Joint  Modeling  and  Simulation  System 

Joint  Requirements  Oversight  Council 

Joint  Simulation  System 

Joint  Technical  Architecture 

Joint  Warfare  System 

Modeling  and  Simulation 

Multipurpose  Internet  Mail  Extensions 

Measure  of  Effectiveness 

Measure  of  Performance 

Military  Operations  Research  Society 

Multiresolution  Modeling 

Modeling  and  Simulation  Resource  Repository 

National  Broadcasting  Corporation 
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NGO 

NIMA 

NRC 

O&S 

00 

OOTW 

OPLAN 

OPS 

OPTEMPO 

OR 

OSD 

OT 

PA&E 

PDU 

PERSTEMPO 

PM 

POM 

QDR 

R&D 

RTI 

S&T 

SAIC 

SBA 

SEDRIS 

SIMNET 

SIMSCRIPT 

SIMTECH 

SOM 

SPI 

TAFIM 

TOR 

TRM 

UML 

USA 

USACOM 

USAF 

v&v 

VPI 

VRML 


Non-govemmental  Organization 

National  Image  and  Mapping  Agency 

National  Research  Council 

Operations  and  Support 

Object-oriented 

Operations  Other  Than  War 

Operations  Plan 

Operational  Position  Standards 

Operations  tempo 

Operations  Research 

Office  of  the  Secretary  of  Defense 

Operational  Test 

Program  Analysis  and  Evaluation 

Protocol  Data  Unit 

Personnel  tempo 

Program  Manager 

Program  Objective  Memoranda 

Quadrennial  Defense  Review 

Research  and  Development 

Run  Time  Infrastructure 

Science  and  Technology 

Science  Application  International  Corporation 

Simulation  Based  Acquisition 

Synthetic  Environment  Data  Representation  and  Interchange  Specification 

Simulation  Network 

A  simulation  language 

Simulation  Technology 

Simulation  Object  Model 

Strategic  Perspectives,  Inc. 

Technical  Architecture  for  Information  Management 

Terms  of  Reference 

Technical  Reference  Model 

Unified  Modeling  Language 

United  States  Army 

US  Atlantic  Command 

United  States  Air  Force 

Validation  and  Verification 

Virginia  Polytechnic  Institute 

Virtual  Reality  Modeling  Language 
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VTC 

Video  Teleconferencing 

VV&A 

Verification,  Validation,  and  Accreditation 

vv&c 

Verification,  Validation,  and  Certification 

WG 

Working  Group 

WMD 

Weapons  of  Mass  Destruction 
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